[GitHub] [airflow] jedcunningham merged pull request #29668: Add Maciej Obuchowski to triage to help with AIP-53 issues

2023-02-21 Thread via GitHub
jedcunningham merged PR #29668: URL: https://github.com/apache/airflow/pull/29668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] jedcunningham opened a new pull request, #29668: Add Maciej Obuchowski to triage to help with AIP-53 issues

2023-02-21 Thread via GitHub
jedcunningham opened a new pull request, #29668: URL: https://github.com/apache/airflow/pull/29668 @mobuchowski is going to help manage issues for AIP-53. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [airflow] joshowen closed pull request #28265: ability to set referenceId in ECSRunTaskOperator

2023-02-21 Thread via GitHub
joshowen closed pull request #28265: ability to set referenceId in ECSRunTaskOperator URL: https://github.com/apache/airflow/pull/28265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [airflow] potiuk commented on pull request #29665: Quarantine `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
potiuk commented on PR #29665: URL: https://github.com/apache/airflow/pull/29665#issuecomment-1438838184 cc: @mhenc I think this one needs to be looked at :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[airflow] branch main updated: Quarantine `test_cli_internal_api_background` (#29665)

2023-02-21 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 66a8d102fc Quarantine `test_cli_internal_api_backgr

[GitHub] [airflow] potiuk commented on pull request #29665: Quarantine `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
potiuk commented on PR #29665: URL: https://github.com/apache/airflow/pull/29665#issuecomment-1438834717 Merged quickly to unblock main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [airflow] potiuk merged pull request #29665: Quarantine `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
potiuk merged PR #29665: URL: https://github.com/apache/airflow/pull/29665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] ashb commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
ashb commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438827099 An option: have a button on the New Connection form that gives "Copy as URI". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] r-richmond commented on pull request #29644: Remove <2.0.0 limit on google-cloud-bigtable

2023-02-21 Thread via GitHub
r-richmond commented on PR #29644: URL: https://github.com/apache/airflow/pull/29644#issuecomment-1438801881 Latest failure looks like another pipeline issue. >/home/runner/work/_temp/5e427642-8610-4b5e-8e04-24f61c20032d.sh: line 1: 4290 Killed breeze testing tests -

[GitHub] [airflow] frankcash opened a new issue, #29666: Multiple Mount Points for Hashicorp Vault Back-end

2023-02-21 Thread via GitHub
frankcash opened a new issue, #29666: URL: https://github.com/apache/airflow/issues/29666 ### Description Support mounting to multiple namespaces with the Hashicorp Vault Secrets Back-end ### Use case/motivation As a data engineer I wish to utilize secrets stored in mult

[GitHub] [airflow] Taragolis opened a new pull request, #29665: Quarantine `test_cli_internal_api_background`

2023-02-21 Thread via GitHub
Taragolis opened a new pull request, #29665: URL: https://github.com/apache/airflow/pull/29665 This tests failed due to timeout (3.5 minutes) in main, temporary quarantine it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [airflow] iJanki-gr closed issue #29426: Mounting all volume in init/side containers breaks my setup (introduced in PR 27420)

2023-02-21 Thread via GitHub
iJanki-gr closed issue #29426: Mounting all volume in init/side containers breaks my setup (introduced in PR 27420) URL: https://github.com/apache/airflow/issues/29426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] potiuk commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
potiuk commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438765361 closing as this is not very likely to be implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [airflow] potiuk commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
potiuk commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438764607 correct. Passwords in the UI are "write only" for that very reason. We are not supposed to read them via UI. And we also have all the - easy - ways to get the URL either via a small

[GitHub] [airflow] potiuk closed issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
potiuk closed issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI URL: https://github.com/apache/airflow/issues/29656 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] fritz-astronomer commented on pull request #29599: fix do_xcom_push=False bug in SnowflakeOperator

2023-02-21 Thread via GitHub
fritz-astronomer commented on PR #29599: URL: https://github.com/apache/airflow/pull/29599#issuecomment-1438761089 Oh! Brilliant! Just short circuit and skip all the rest. That makes sense. Not sure why I was worried that method might be needed for side-effects other than xcom -- This is

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-21 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1113290164 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[GitHub] [airflow] syun64 opened a new issue, #29663: Option to Disable High Cardinality Metrics on Statsd

2023-02-21 Thread via GitHub
syun64 opened a new issue, #29663: URL: https://github.com/apache/airflow/issues/29663 ### Description With recent PRs enabling tags-support on Statsd metrics, we gained a deeper understanding into the issue of publishing high cardinality metrics. Through this issue, I hope to facili

[GitHub] [airflow] vincbeck commented on a diff in pull request #29581: AWS system test sagemaker-endpoint: archive logs

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #29581: URL: https://github.com/apache/airflow/pull/29581#discussion_r1113261654 ## tests/system/providers/amazon/aws/example_sagemaker_endpoint.py: ## @@ -88,6 +88,11 @@ def delete_endpoint(endpoint_name): boto3.client("sagemaker").delete_e

[GitHub] [airflow] vincbeck commented on a diff in pull request #29581: AWS system test sagemaker-endpoint: archive logs

2023-02-21 Thread via GitHub
vincbeck commented on code in PR #29581: URL: https://github.com/apache/airflow/pull/29581#discussion_r1113261654 ## tests/system/providers/amazon/aws/example_sagemaker_endpoint.py: ## @@ -88,6 +88,11 @@ def delete_endpoint(endpoint_name): boto3.client("sagemaker").delete_e

[GitHub] [airflow] antonio-antuan commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-21 Thread via GitHub
antonio-antuan commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1438707930 yes, the same warnings :cry: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-21 Thread via GitHub
ashb commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1113247672 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._session

[GitHub] [airflow] vchiapaikeo commented on issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-21 Thread via GitHub
vchiapaikeo commented on issue #29607: URL: https://github.com/apache/airflow/issues/29607#issuecomment-1438702684 https://github.com/apache/airflow/pull/28942 still looks good. https://user-images.githubusercontent.com/9200263/220392003-9ee1fda4-0167-497e-8531-cbd0363530fb.png";>

[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-21 Thread via GitHub
ashb commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1113244622 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._session

[GitHub] [airflow] ashb commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-21 Thread via GitHub
ashb commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1113233527 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -311,20 +311,32 @@ def _get_web_identity_credential_fetcher( ) -> botocore.credentials.AssumeRoleWithWebIde

[GitHub] [airflow] bugraoz93 commented on issue #29315: AIP-44 Migrate BaseJob.heartbeat to InternalAPI

2023-02-21 Thread via GitHub
bugraoz93 commented on issue #29315: URL: https://github.com/apache/airflow/issues/29315#issuecomment-1438681142 Thank you! 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [airflow] michaelmicheal commented on a diff in pull request #29433: Add dataset update endpoint

2023-02-21 Thread via GitHub
michaelmicheal commented on code in PR #29433: URL: https://github.com/apache/airflow/pull/29433#discussion_r1113218259 ## airflow/datasets/manager.py: ## @@ -55,23 +61,33 @@ def register_dataset_change( dataset_model = session.query(DatasetModel).filter(DatasetModel.ur

[GitHub] [airflow] michaelmicheal commented on a diff in pull request #29433: Add dataset update endpoint

2023-02-21 Thread via GitHub
michaelmicheal commented on code in PR #29433: URL: https://github.com/apache/airflow/pull/29433#discussion_r1113218259 ## airflow/datasets/manager.py: ## @@ -55,23 +61,33 @@ def register_dataset_change( dataset_model = session.query(DatasetModel).filter(DatasetModel.ur

[GitHub] [airflow] renzo-sanchez-h commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
renzo-sanchez-h commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1438661458 @ephraimbuddy hello, yes, I edited the code above you can activate both subflows by uncommenting PGP_OPERATION, this for testing the other case. -- This is an automate

[airflow] branch main updated: docs: fix typo (#29658)

2023-02-21 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 37a317286a docs: fix typo (#29658) 37a317286a is d

[GitHub] [airflow] eladkal merged pull request #29658: docs: fix typo docs of remote logging

2023-02-21 Thread via GitHub
eladkal merged PR #29658: URL: https://github.com/apache/airflow/pull/29658 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[GitHub] [airflow] Taragolis commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
Taragolis commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438653287 AFAIK it is not possible retrieve `password` field in Webserver/REST API, potentially for security reason -- This is an automated message from the Apache Git Service. To respon

[GitHub] [airflow] arjunanan6 opened a new issue, #29662: Audit Log is unclear when using Azure AD login

2023-02-21 Thread via GitHub
arjunanan6 opened a new issue, #29662: URL: https://github.com/apache/airflow/issues/29662 ### Apache Airflow version 2.5.1 ### What happened We're using an Azure OAUTH based login in our Airflow implementation, and everything works great. This is more of a visual proble

[GitHub] [airflow] Taragolis opened a new pull request, #29661: Disable unixodbc and related packages from Microsoft APT repo

2023-02-21 Thread via GitHub
Taragolis opened a new pull request, #29661: URL: https://github.com/apache/airflow/pull/29661 Disable `unixodbc` and related packages from https://packages.microsoft.com/ and use packages from Debian repo instead. - https://github.com/microsoft/linux-package-repositories/issues/36#i

[GitHub] [airflow] romibuzi opened a new pull request, #29659: AWS Glue job hook: Make s3_bucket parameter optional

2023-02-21 Thread via GitHub
romibuzi opened a new pull request, #29659: URL: https://github.com/apache/airflow/pull/29659 Solves https://github.com/apache/airflow/issues/29423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] KennethanCeyer opened a new pull request, #29658: docs: fix typo docs of remote logging

2023-02-21 Thread via GitHub
KennethanCeyer opened a new pull request, #29658: URL: https://github.com/apache/airflow/pull/29658 There's a typo in the Airflow remote logging documentation where "eigion" is mistakenly used instead of "region." This error could cause confusion among readers, and it's important to

[GitHub] [airflow] potiuk commented on issue #25297: on_failure_callback is not called when task is terminated externally

2023-02-21 Thread via GitHub
potiuk commented on issue #25297: URL: https://github.com/apache/airflow/issues/25297#issuecomment-1438588037 It was closed, because we did not have enough evidences and could not reproduce it. And it's really strange to be sure that it is the same bug. It might look and behave similarly an

[GitHub] [airflow] hliu47 opened a new issue, #25297: on_failure_callback is not called when task is terminated externally

2023-02-21 Thread via GitHub
hliu47 opened a new issue, #25297: URL: https://github.com/apache/airflow/issues/25297 ### Apache Airflow version 2.2.5 ### What happened `on_failure_callback` is not called when task is terminated externally. A similar issue was reported in [#14422](https://github.co

[GitHub] [airflow] pankajastro commented on a diff in pull request #29642: Migrate tests in google/cloud/sensors from unittest to pytest

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #29642: URL: https://github.com/apache/airflow/pull/29642#discussion_r1113134255 ## tests/providers/google/cloud/sensors/test_bigquery.py: ## @@ -37,7 +37,7 @@ TEST_IMPERSONATION_CHAIN = ["ACCOUNT_1", "ACCOUNT_2", "ACCOUNT_3"] -class Test

[GitHub] [airflow] Oduig commented on issue #29461: Timeout in DatabricksSubmitRunOperator causes callbacks to be invoked twice

2023-02-21 Thread via GitHub
Oduig commented on issue #29461: URL: https://github.com/apache/airflow/issues/29461#issuecomment-1438575233 If I use a `BashOperator` instead, the failure callback is only invoked once. I checked the stack traces by using `traceback.extract_stack()`. It could be related to Composer, we can

[GitHub] [airflow] seub commented on issue #25297: on_failure_callback is not called when task is terminated externally

2023-02-21 Thread via GitHub
seub commented on issue #25297: URL: https://github.com/apache/airflow/issues/25297#issuecomment-1438572980 @potiuk This issue was automatically closed by a bot without being solved. I am confirming that it is in fact not solved, having the exact same behavior on 2.5.1. I'm only reporting

[GitHub] [airflow] syun64 commented on pull request #28961: Emit DataDog statsd metrics with metadata tags

2023-02-21 Thread via GitHub
syun64 commented on PR #28961: URL: https://github.com/apache/airflow/pull/28961#issuecomment-1438566759 Thank you for that reference @potiuk - I think we can lean in on the fact that OTEL is also trying better document the problems high cardinality metrics pose to the users and justify imp

[GitHub] [airflow] planoe commented on pull request #29357: Template extra volumes in helm chart

2023-02-21 Thread via GitHub
planoe commented on PR #29357: URL: https://github.com/apache/airflow/pull/29357#issuecomment-1438566460 > I think this one needs documentation explaining usage of this feature. Otherwise it won't be discoverable. Fair point. I suggest to add a comment/example above the impacted field

[GitHub] [airflow] josh-fell commented on a diff in pull request #29548: Syedahsn/ec2 create terminate operators

2023-02-21 Thread via GitHub
josh-fell commented on code in PR #29548: URL: https://github.com/apache/airflow/pull/29548#discussion_r1113091829 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -116,3 +116,139 @@ def execute(self, context: Context): target_state="stopped", ch

[GitHub] [airflow] phanikumv commented on a diff in pull request #29300: Add deferrable BatchOperator

2023-02-21 Thread via GitHub
phanikumv commented on code in PR #29300: URL: https://github.com/apache/airflow/pull/29300#discussion_r1113081966 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -865,3 +868,85 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[GitHub] [airflow] josh-fell commented on a diff in pull request #29549: Fix and augment `check-for-inclusive-language` CI check

2023-02-21 Thread via GitHub
josh-fell commented on code in PR #29549: URL: https://github.com/apache/airflow/pull/29549#discussion_r1113079693 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -240,7 +240,7 @@ def get_pandas_df( query. The DbApiHook method must be overridden because Panda

[GitHub] [airflow] josh-fell commented on a diff in pull request #29549: Fix and augment `check-for-inclusive-language` CI check

2023-02-21 Thread via GitHub
josh-fell commented on code in PR #29549: URL: https://github.com/apache/airflow/pull/29549#discussion_r1113079693 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -240,7 +240,7 @@ def get_pandas_df( query. The DbApiHook method must be overridden because Panda

[GitHub] [airflow] potiuk commented on pull request #29143: Demonstrate usage of the PythonSensor

2023-02-21 Thread via GitHub
potiuk commented on PR #29143: URL: https://github.com/apache/airflow/pull/29143#issuecomment-1438506435 > We should separate purposes and store code for docs separate from tests, i.e. in the /docs folder. Since that would be a bigger change, happy to close this PR for now and use it to dem

[GitHub] [airflow] pankajastro opened a new pull request, #29657: Add `wait_for_completion` param in `RedshiftCreateClusterOperator`

2023-02-21 Thread via GitHub
pankajastro opened a new pull request, #29657: URL: https://github.com/apache/airflow/pull/29657 Add `wait_for_completion` param in RedshiftCreateClusterOperator this will enable me to use a single operator to make a cluster in the available state --- **^ Add meaningful descrip

[GitHub] [airflow] BasPH commented on pull request #29143: Demonstrate usage of the PythonSensor

2023-02-21 Thread via GitHub
BasPH commented on PR #29143: URL: https://github.com/apache/airflow/pull/29143#issuecomment-1438485869 We should think better about the user experience. Currently, example DAGs are displayed by default. I've never met a user that actually wants that. The first question is usually "ho

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
boring-cyborg[bot] commented on issue #29656: URL: https://github.com/apache/airflow/issues/29656#issuecomment-1438479480 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] manmeetkaur opened a new issue, #29656: Generate URI/JSON for a connection from the Airflow Connection UI

2023-02-21 Thread via GitHub
manmeetkaur opened a new issue, #29656: URL: https://github.com/apache/airflow/issues/29656 ### Description Ability to generate the Connection String in either URI or JSON Format from the Connection UX on the Airflow UI. ### Use case/motivation Many a times, have seen Da

[GitHub] [airflow] potiuk commented on a diff in pull request #29355: AIP-44 Support TaskInstance serialization/deserialization.

2023-02-21 Thread via GitHub
potiuk commented on code in PR #29355: URL: https://github.com/apache/airflow/pull/29355#discussion_r1113026086 ## airflow/serialization/serialized_objects.py: ## @@ -502,6 +504,8 @@ def deserialize(cls, encoded_var: Any) -> Any: return Dataset(**var) elif

[GitHub] [airflow] potiuk commented on a diff in pull request #29355: AIP-44 Support TaskInstance serialization/deserialization.

2023-02-21 Thread via GitHub
potiuk commented on code in PR #29355: URL: https://github.com/apache/airflow/pull/29355#discussion_r1113026086 ## airflow/serialization/serialized_objects.py: ## @@ -502,6 +504,8 @@ def deserialize(cls, encoded_var: Any) -> Any: return Dataset(**var) elif

[GitHub] [airflow] potiuk commented on a diff in pull request #28256: Include full path to Python files under zip path while clearing import errors.

2023-02-21 Thread via GitHub
potiuk commented on code in PR #28256: URL: https://github.com/apache/airflow/pull/28256#discussion_r1113025147 ## airflow/dag_processing/manager.py: ## @@ -782,7 +782,11 @@ def clear_nonexistent_import_errors(file_paths: list[str] | None, session=NEW_SE """ q

[GitHub] [airflow] potiuk commented on a diff in pull request #28256: Include full path to Python files under zip path while clearing import errors.

2023-02-21 Thread via GitHub
potiuk commented on code in PR #28256: URL: https://github.com/apache/airflow/pull/28256#discussion_r1113020259 ## airflow/dag_processing/manager.py: ## @@ -782,7 +782,11 @@ def clear_nonexistent_import_errors(file_paths: list[str] | None, session=NEW_SE """ q

[GitHub] [airflow] potiuk commented on issue #27772: Unable to get dag logs from Web UI

2023-02-21 Thread via GitHub
potiuk commented on issue #27772: URL: https://github.com/apache/airflow/issues/27772#issuecomment-1438433519 Yep. Thanks for confirming it! https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable explains `get_host_ip_address` as a possible option t

[GitHub] [airflow] pschoenmakers-ah commented on issue #29650: Post installation upgrades fail for postgres

2023-02-21 Thread via GitHub
pschoenmakers-ah commented on issue #29650: URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438429597 This was indeed the root cause but it was not really clear how to resolve. In the end I got it going like this: - create a values file - do a --dry-run install t

[GitHub] [airflow] ephraimbuddy commented on issue #29199: TaskFlow AirflowSkipException causes downstream step to fail when multiple_outputs is true

2023-02-21 Thread via GitHub
ephraimbuddy commented on issue #29199: URL: https://github.com/apache/airflow/issues/29199#issuecomment-1438413484 @renzo-sanchez-h please can you write a simple dag to reproduce this scenario. The dag above cannot run outside your environment. Have tweaked it but I can't reproduce the iss

[GitHub] [airflow] pankajastro commented on a diff in pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #28850: URL: https://github.com/apache/airflow/pull/28850#discussion_r1112992742 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -843,3 +848,82 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[GitHub] [airflow] Taragolis commented on pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-21 Thread via GitHub
Taragolis commented on PR #29635: URL: https://github.com/apache/airflow/pull/29635#issuecomment-1438406620 Looks nice, there are two things left 1. Static Checks failed, I would recommend configure locally pre-commit hooks, see [Static code checks](https://github.com/apache/airflow/blob

[GitHub] [airflow] kobethuwis commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3

2023-02-21 Thread via GitHub
kobethuwis commented on issue #29393: URL: https://github.com/apache/airflow/issues/29393#issuecomment-1438402879 Due to this issue, I tried understanding the way Airflow is (remotely) logging tasks. I am using S3 for remote log storage, but would like to access live logs for running tasks

[GitHub] [airflow] mhenc commented on a diff in pull request #29355: AIP-44 Support TaskInstance serialization/deserialization.

2023-02-21 Thread via GitHub
mhenc commented on code in PR #29355: URL: https://github.com/apache/airflow/pull/29355#discussion_r1112983708 ## airflow/serialization/serialized_objects.py: ## @@ -502,6 +504,8 @@ def deserialize(cls, encoded_var: Any) -> Any: return Dataset(**var) elif t

[GitHub] [airflow] mhenc commented on a diff in pull request #29513: AIP-44 Migrate TaskInstance.check_and_change_state_before_execution to Internal API

2023-02-21 Thread via GitHub
mhenc commented on code in PR #29513: URL: https://github.com/apache/airflow/pull/29513#discussion_r1112985868 ## airflow/models/taskinstance.py: ## @@ -1207,9 +1208,11 @@ def get_dagrun(self, session: Session = NEW_SESSION) -> DagRun: return dr +@staticmethod

[GitHub] [airflow] mhenc commented on a diff in pull request #29355: AIP-44 Support TaskInstance serialization/deserialization.

2023-02-21 Thread via GitHub
mhenc commented on code in PR #29355: URL: https://github.com/apache/airflow/pull/29355#discussion_r1112983708 ## airflow/serialization/serialized_objects.py: ## @@ -502,6 +504,8 @@ def deserialize(cls, encoded_var: Any) -> Any: return Dataset(**var) elif t

[GitHub] [airflow] mhenc commented on a diff in pull request #29355: AIP-44 Support TaskInstance serialization/deserialization.

2023-02-21 Thread via GitHub
mhenc commented on code in PR #29355: URL: https://github.com/apache/airflow/pull/29355#discussion_r1112983708 ## airflow/serialization/serialized_objects.py: ## @@ -502,6 +504,8 @@ def deserialize(cls, encoded_var: Any) -> Any: return Dataset(**var) elif t

[GitHub] [airflow] tirkarthi commented on a diff in pull request #28256: Include full path to Python files under zip path while clearing import errors.

2023-02-21 Thread via GitHub
tirkarthi commented on code in PR #28256: URL: https://github.com/apache/airflow/pull/28256#discussion_r1112980514 ## airflow/dag_processing/manager.py: ## @@ -782,7 +782,11 @@ def clear_nonexistent_import_errors(file_paths: list[str] | None, session=NEW_SE """

[GitHub] [airflow] kobethuwis commented on issue #27772: Unable to get dag logs from Web UI

2023-02-21 Thread via GitHub
kobethuwis commented on issue #27772: URL: https://github.com/apache/airflow/issues/27772#issuecomment-1438390751 Due to this issue, I tried understanding the way Airflow is logging tasks. > In the Airflow UI, remote logs take precedence over local logs when remote logging is enabled.

[GitHub] [airflow] pankajastro commented on a diff in pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #28850: URL: https://github.com/apache/airflow/pull/28850#discussion_r1112978007 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -843,3 +848,82 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[GitHub] [airflow] pankajastro commented on a diff in pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #28850: URL: https://github.com/apache/airflow/pull/28850#discussion_r1112975396 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -843,3 +848,82 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[GitHub] [airflow] pankajastro commented on a diff in pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #28850: URL: https://github.com/apache/airflow/pull/28850#discussion_r1112971388 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -843,3 +848,82 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[airflow] branch revert-29408-docker-compose-change-example updated (37aef76496 -> a8124e3bbe)

2023-02-21 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git omit 37aef76496 Revert "Improve health checks in example docker-compose and clarif

[GitHub] [airflow] potiuk commented on a diff in pull request #29513: AIP-44 Migrate TaskInstance.check_and_change_state_before_execution to Internal API

2023-02-21 Thread via GitHub
potiuk commented on code in PR #29513: URL: https://github.com/apache/airflow/pull/29513#discussion_r1112952981 ## airflow/models/taskinstance.py: ## @@ -1207,9 +1208,11 @@ def get_dagrun(self, session: Session = NEW_SESSION) -> DagRun: return dr +@staticmethod

[GitHub] [airflow] V0lantis commented on issue #29654: Not able to specify redis as a result backend

2023-02-21 Thread via GitHub
V0lantis commented on issue #29654: URL: https://github.com/apache/airflow/issues/29654#issuecomment-1438332301 I would like to add that there is no `job_type` for the dag-processor and therefore, the liveness probe is pointless since it cannot find any job in the database -- This is an

[GitHub] [airflow] V0lantis opened a new issue, #29654: Not able to specify redis as a result backend

2023-02-21 Thread via GitHub
V0lantis opened a new issue, #29654: URL: https://github.com/apache/airflow/issues/29654 ### Official Helm Chart version 1.8.0 (latest released) ### Apache Airflow version 2.4.3 ### Kubernetes Version v1.21.14-eks ### Helm Chart configuration `

[GitHub] [airflow] potiuk commented on pull request #29653: separate stale jobs to dedicate jobs in different intervals

2023-02-21 Thread via GitHub
potiuk commented on PR #29653: URL: https://github.com/apache/airflow/pull/29653#issuecomment-1438241731 or rather "needs:" in the github actions convention -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] potiuk commented on pull request #29653: separate stale jobs to dedicate jobs in different intervals

2023-02-21 Thread via GitHub
potiuk commented on PR #29653: URL: https://github.com/apache/airflow/pull/29653#issuecomment-1438238082 Wouldn't that be better to leave them in one workflow and add "depends on"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [airflow] eladkal opened a new pull request, #29653: separate stale jobs to dedicate jobs in different intervals

2023-02-21 Thread via GitHub
eladkal opened a new pull request, #29653: URL: https://github.com/apache/airflow/pull/29653 to avoid cases where jobs might overlap. better to let first scan to over before starting the other. --- **^ Add meaningful description above** Read the **[Pull Request Guid

[GitHub] [airflow] uranusjr commented on issue #20762: Create documentation on @task_group

2023-02-21 Thread via GitHub
uranusjr commented on issue #20762: URL: https://github.com/apache/airflow/issues/20762#issuecomment-1438207683 This was finished in #28001. I totally forgot this issue exists, thanks for the reminder! -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] uranusjr closed issue #20762: Create documentation on @task_group

2023-02-21 Thread via GitHub
uranusjr closed issue #20762: Create documentation on @task_group URL: https://github.com/apache/airflow/issues/20762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[airflow] branch main updated (1edaddbb1c -> 0018b94a4a)

2023-02-21 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1edaddbb1c Avoid unneeded Connexion constructs (#29631) add 0018b94a4a introduce dag processor job (#28799) No ne

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28799: introduce dag processor job fix #27140

2023-02-21 Thread via GitHub
boring-cyborg[bot] commented on PR #28799: URL: https://github.com/apache/airflow/pull/28799#issuecomment-1438180443 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] potiuk merged pull request #28799: introduce dag processor job fix #27140

2023-02-21 Thread via GitHub
potiuk merged PR #28799: URL: https://github.com/apache/airflow/pull/28799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] mhenc commented on pull request #28799: introduce dag processor job fix #27140

2023-02-21 Thread via GitHub
mhenc commented on PR #28799: URL: https://github.com/apache/airflow/pull/28799#issuecomment-1438175707 Thank you for looking int that. This change looks good for me. TBH I haven't tested it with Helm (as we use other way of deploying workloads to K8s). -- This is an automated

[GitHub] [airflow] pankajastro commented on a diff in pull request #28850: Add deferrable mode in RedshiftPauseClusterOperator

2023-02-21 Thread via GitHub
pankajastro commented on code in PR #28850: URL: https://github.com/apache/airflow/pull/28850#discussion_r1112816670 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -843,3 +848,82 @@ def _parse_s3_config(config_file_name: str, config_format: str | None = "boto",

[GitHub] [airflow] tanelk commented on pull request #28943: Add handling logic on CeleryExecutor to reschedule task stuck in queued status

2023-02-21 Thread via GitHub
tanelk commented on PR #28943: URL: https://github.com/apache/airflow/pull/28943#issuecomment-1438153269 Doesn't the `stalled_task_timeout` configuration in the `[celery]` block do exactly this? It defaults to 0, meaning it is disabled. -- This is an automated message from the Apache Git

[GitHub] [airflow] VinceLegendre commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
VinceLegendre commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112720178 ## airflow/providers/google/cloud/hooks/cloud_run.py: ## @@ -0,0 +1,245 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribut

[airflow] branch main updated (f51742d20b -> 1edaddbb1c)

2023-02-21 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f51742d20b Don't push secret in XCOM in BigQueryCreateDataTransferOperator (#29348) add 1edaddbb1c Avoid unneed

[GitHub] [airflow] uranusjr merged pull request #29631: Avoid unneeded Connexion constructs

2023-02-21 Thread via GitHub
uranusjr merged PR #29631: URL: https://github.com/apache/airflow/pull/29631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] Taragolis commented on a diff in pull request #29616: Refactor docker-compose quick start test

2023-02-21 Thread via GitHub
Taragolis commented on code in PR #29616: URL: https://github.com/apache/airflow/pull/29616#discussion_r1112728077 ## docker_tests/test_docker_compose_quick_start.py: ## @@ -114,53 +104,60 @@ def wait_for_terminal_dag_state(dag_id, dag_run_id): break -def test_t

[GitHub] [airflow] VinceLegendre commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
VinceLegendre commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112720178 ## airflow/providers/google/cloud/hooks/cloud_run.py: ## @@ -0,0 +1,245 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribut

[GitHub] [airflow] raphaelauv commented on pull request #29498: add missing read for K8S config file from conn in deferred `KubernetesPodOperator`

2023-02-21 Thread via GitHub
raphaelauv commented on PR #29498: URL: https://github.com/apache/airflow/pull/29498#issuecomment-1438040543 Loading the config file from the env KUBECONFIG is deprecated in latest provider version -- This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [airflow] potiuk commented on issue #29366: default_args feature incompatible with Dynamic Task Mapping

2023-02-21 Thread via GitHub
potiuk commented on issue #29366: URL: https://github.com/apache/airflow/issues/29366#issuecomment-1438031619 > Overriding params in expand which are defined via default_args would be intended behaviour and in line with the behaviour of normal operators, wouldn't be? Yes. By "worry"

[GitHub] [airflow] potiuk commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
potiuk commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112703777 ## airflow/providers/google/cloud/hooks/cloud_run.py: ## @@ -0,0 +1,245 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor lice

[GitHub] [airflow] m1racoli commented on issue #29366: default_args feature incompatible with Dynamic Task Mapping

2023-02-21 Thread via GitHub
m1racoli commented on issue #29366: URL: https://github.com/apache/airflow/issues/29366#issuecomment-1438025543 > my only worry is whether expand would override them if they are also specified there Overriding params in expand which are defined via default_args would be intended behaviou

[GitHub] [airflow] joshuaghezzi commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-21 Thread via GitHub
joshuaghezzi commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1438023127 Thanks a lot for the input @andrewgodwin and @potiuk. Happy to close this one as won't do. -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [airflow] joshuaghezzi closed issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-21 Thread via GitHub
joshuaghezzi closed issue #29648: Optionally exclude task deferral time from overall runtime of the task URL: https://github.com/apache/airflow/issues/29648 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] potiuk commented on issue #29650: Post installation upgrades fail for postgres

2023-02-21 Thread via GitHub
potiuk commented on issue #29650: URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438019362 This ie expected - se release notes. Built-in postgres is "development" feature only and you should not rely on it as working without breaking changes. You need to fix things manual

[GitHub] [airflow] potiuk closed issue #29650: Post installation upgrades fail for postgres

2023-02-21 Thread via GitHub
potiuk closed issue #29650: Post installation upgrades fail for postgres URL: https://github.com/apache/airflow/issues/29650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] VinceLegendre commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-21 Thread via GitHub
VinceLegendre commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112696162 ## airflow/providers/google/cloud/hooks/cloud_run.py: ## @@ -0,0 +1,245 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribut

<    1   2   3   >