[GitHub] [airflow] potiuk commented on a diff in pull request #32608: Add an option to use a direct DB connection in KEDA when pgbouncer is enabled

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32608: URL: https://github.com/apache/airflow/pull/32608#discussion_r1264347266 ## chart/values.yaml: ## @@ -562,6 +562,10 @@ workers: AND queue != '{{ .Values.config.celery_kubernetes_executor.kubernetes_queue }}' {{- end }} +

[airflow] branch main updated: Introduce Heartbeat Parameter to Allow Per-LocalTaskJob Configuration (#32313)

2023-07-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9b466bd13d Introduce Heartbeat Parameter to Allow P

[GitHub] [airflow] potiuk merged pull request #32313: Introduce Heartbeat Parameter to Allow Per-LocalTaskJob Configuration

2023-07-14 Thread via GitHub
potiuk merged PR #32313: URL: https://github.com/apache/airflow/pull/32313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #32313: Introduce Heartbeat Parameter to Allow Per-LocalTaskJob Configuration

2023-07-14 Thread via GitHub
potiuk commented on PR #32313: URL: https://github.com/apache/airflow/pull/32313#issuecomment-1636691808 LGTM. Good thinking about back-compatibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] potiuk commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636690972 And called for somone to implement it :) . Time to become less SPOF for the whole CI for me :) -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [airflow] potiuk opened a new issue, #32620: Add separate test job to run only time-bound tests in isolation

2023-07-14 Thread via GitHub
potiuk opened a new issue, #32620: URL: https://github.com/apache/airflow/issues/32620 ### Body There are few tests that have some "timing" component - i.e they depend on how fast they are run. This might lead to flaky tests when the test are run in parallel to some other tests in v

[airflow] branch main updated: Fix Failing ES Remote Logging (#32438)

2023-07-14 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 41e611929e Fix Failing ES Remote Logging (#32438)

[GitHub] [airflow] eladkal merged pull request #32438: Fix Failing ES Remote Logging

2023-07-14 Thread via GitHub
eladkal merged PR #32438: URL: https://github.com/apache/airflow/pull/32438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[airflow] branch main updated: Add a test to catch regressions in cli run time (#32612)

2023-07-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7fa524deb3 Add a test to catch regressions in cli r

[GitHub] [airflow] potiuk merged pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk merged PR #32612: URL: https://github.com/apache/airflow/pull/32612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow-site] branch add-documentation-2023-07-12 updated (16e00b60b9 -> 0a7222668e)

2023-07-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-documentation-2023-07-12 in repository https://gitbox.apache.org/repos/asf/airflow-site.git omit 16e00b60b9 Add documentation for packages - 2023-07-12 add 0a7222668e Add documentati

[GitHub] [airflow] shubham22 commented on a diff in pull request #32591: D205 Support - Utils

2023-07-14 Thread via GitHub
shubham22 commented on code in PR #32591: URL: https://github.com/apache/airflow/pull/32591#discussion_r1264306575 ## airflow/utils/operator_helpers.py: ## @@ -199,6 +199,8 @@ def determine_kwargs( def make_kwargs_callable(func: Callable[..., R]) -> Callable[..., R]: """

[GitHub] [airflow] shubham22 commented on a diff in pull request #32591: D205 Support - Utils

2023-07-14 Thread via GitHub
shubham22 commented on code in PR #32591: URL: https://github.com/apache/airflow/pull/32591#discussion_r1264305521 ## airflow/utils/task_group.py: ## @@ -407,6 +400,8 @@ def upstream_join_id(self) -> str: @property def downstream_join_id(self) -> str: """ +

[GitHub] [airflow] shubham22 commented on a diff in pull request #32591: D205 Support - Utils

2023-07-14 Thread via GitHub
shubham22 commented on code in PR #32591: URL: https://github.com/apache/airflow/pull/32591#discussion_r1264305464 ## airflow/utils/task_group.py: ## @@ -398,6 +389,8 @@ def child_id(self, label): @property def upstream_join_id(self) -> str: """ +# TOD

[GitHub] [airflow] dirrao commented on pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-14 Thread via GitHub
dirrao commented on PR #32249: URL: https://github.com/apache/airflow/pull/32249#issuecomment-1636640787 @hussein-awala Thanks for sharing the example. The reason for this issue is schedulers are subscribed to their worker pod events based on the scheduler id label filter i.e. ( kwargs

[GitHub] [airflow] shubham22 commented on a diff in pull request #32591: D205 Support - Utils

2023-07-14 Thread via GitHub
shubham22 commented on code in PR #32591: URL: https://github.com/apache/airflow/pull/32591#discussion_r1264296007 ## airflow/utils/log/logging_mixin.py: ## @@ -174,15 +172,18 @@ def flush(self): def isatty(self): """ Returns False to indicate the fd is no

[GitHub] [airflow] hussein-awala commented on pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-14 Thread via GitHub
hussein-awala commented on PR #32249: URL: https://github.com/apache/airflow/pull/32249#issuecomment-1636610244 Good investigation, I just created a simple pod for testing: ```yaml apiVersion: v1 kind: Pod metadata: name: test-pod labels: airflow-worker: scheduler

[GitHub] [airflow] github-actions[bot] commented on pull request #29771: [WIP] Deprecate `execution_date` in the Airflow REST API

2023-07-14 Thread via GitHub
github-actions[bot] commented on PR #29771: URL: https://github.com/apache/airflow/pull/29771#issuecomment-1636577395 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #30331: AIP-44 Migrate MetastoreBackend to Internal API

2023-07-14 Thread via GitHub
github-actions[bot] commented on PR #30331: URL: https://github.com/apache/airflow/pull/30331#issuecomment-1636577364 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #31008: Add startup_check_interval_seconds to PodManager's await_pod_start

2023-07-14 Thread via GitHub
github-actions[bot] commented on PR #31008: URL: https://github.com/apache/airflow/pull/31008#issuecomment-1636577343 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] ferruzzi commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
ferruzzi commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636573402 That test was green as well,just re-triggered all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [airflow] ferruzzi commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-14 Thread via GitHub
ferruzzi commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1636572812 @Adaverse - Anything else to add? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [airflow] hussein-awala commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-14 Thread via GitHub
hussein-awala commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1636558751 I'll try to reproduce the issue. Which version of `confluent-kafka` and Kafka (in the cluster) are you using? -- This is an automated message from the Apache Git Service. T

[GitHub] [airflow] o-nikolas commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636540913 Triggering another round of tests, the last set were green -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32619: Make dag_run join as lazyload by default in TaskInstance

2023-07-14 Thread via GitHub
boring-cyborg[bot] commented on PR #32619: URL: https://github.com/apache/airflow/pull/32619#issuecomment-1636536293 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] haoch opened a new pull request, #32619: Make dag_run join as lazyload by default in TaskInstance

2023-07-14 Thread via GitHub
haoch opened a new pull request, #32619: URL: https://github.com/apache/airflow/pull/32619 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for mo

[GitHub] [airflow] o-nikolas commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636525176 > BTW. This is a constant risk - influx of tests that are flaky. I personally think that this is something that is difficult to handle, but it's a different story that we

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-14 Thread via GitHub
ferruzzi commented on code in PR #32498: URL: https://github.com/apache/airflow/pull/32498#discussion_r1264209408 ## tests/system/providers/amazon/aws/example_eventbridge.py: ## @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-14 Thread via GitHub
vandonr-amz commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1264191556 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -18,16 +18,29 @@ import logging import re +import warnings from datetime import datetime from enum

[GitHub] [airflow] potiuk commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1264184077 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -805,10 +805,7 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo ```shell script

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264177557 ## airflow/models/dagrun.py: ## @@ -100,6 +102,29 @@ def _creator_note(val): return DagRunNote(*val) +def _get_task_instance( Review Comment: Why is

[GitHub] [airflow] brokenjacobs commented on pull request #32608: Add an option to use a direct DB connection in KEDA when pgbouncer is enabled

2023-07-14 Thread via GitHub
brokenjacobs commented on PR #32608: URL: https://github.com/apache/airflow/pull/32608#issuecomment-1636447903 I guess I'll PR this once I can test this query after 1.11.0 comes out, but I think this keda query will account for pool restrictions, which the current query does not do: ```

[GitHub] [airflow] potiuk commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636447818 BTW. This is a constant risk - influx of tests that are flaky. I personally think that this is something that is difficult to handle, but it's a different story that we KNOW that a test

[GitHub] [airflow] potiuk commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636442899 Closed /reopen to try it on public runners (possibly we should repeat a few times) -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] potiuk closed pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk closed pull request #32612: Add a test to catch regressions in cli run time URL: https://github.com/apache/airflow/pull/32612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [airflow] potiuk commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1264174126 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-forma

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264140862 ## airflow/utils/context.pyi: ## @@ -37,11 +37,13 @@ from airflow.models.dagrun import DagRun from airflow.models.dataset import DatasetEvent from airflow.models.p

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264133945 ## pyproject.toml: ## @@ -145,7 +145,7 @@ known-third-party = [ # The Pydantic representations of SqlAlchemy Models are not parsed well with Pydantic # when __f

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264132960 ## airflow/models/serialized_dag.py: ## @@ -395,3 +398,18 @@ def get_dag_dependencies(cls, session: Session = NEW_SESSION) -> dict[str, list[ else:

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264131516 ## airflow/ti_deps/deps/prev_dagrun_dep.py: ## @@ -19,6 +19,7 @@ from sqlalchemy import func +from airflow.models import DagRun Review Comment: Same applies

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264130977 ## airflow/ti_deps/deps/prev_dagrun_dep.py: ## @@ -19,6 +19,7 @@ from sqlalchemy import func +from airflow.models import DagRun Review Comment: ```suggestio

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264130731 ## airflow/utils/context.pyi: ## @@ -37,11 +37,13 @@ from airflow.models.dagrun import DagRun from airflow.models.dataset import DatasetEvent from airflow.models.p

[GitHub] [airflow] uranusjr commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1264129531 ## pyproject.toml: ## @@ -145,7 +145,7 @@ known-third-party = [ # The Pydantic representations of SqlAlchemy Models are not parsed well with Pydantic # when __f

[GitHub] [airflow] uranusjr commented on a diff in pull request #32520: allows users to write dag_id and task_id in their national characters, added display name for dag / task

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #32520: URL: https://github.com/apache/airflow/pull/32520#discussion_r1264106427 ## airflow/www/templates/airflow/task_instance.html: ## @@ -25,7 +25,7 @@ -Task Instance: {{ task_id }} +Task Instance: {{display_name}} Review

[GitHub] [airflow] uranusjr commented on a diff in pull request #32520: allows users to write dag_id and task_id in their national characters, added display name for dag / task

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #32520: URL: https://github.com/apache/airflow/pull/32520#discussion_r1264105039 ## airflow/models/dag.py: ## @@ -3380,6 +3388,8 @@ class DagModel(Base): processor_subdir = Column(String(2000), nullable=True) # String representing the o

[GitHub] [airflow] uranusjr commented on a diff in pull request #32520: allows users to write dag_id and task_id in their national characters, added display name for dag / task

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #32520: URL: https://github.com/apache/airflow/pull/32520#discussion_r1264104223 ## airflow/models/dag.py: ## @@ -3380,6 +3388,8 @@ class DagModel(Base): processor_subdir = Column(String(2000), nullable=True) # String representing the o

[GitHub] [airflow] dstandish commented on a diff in pull request #31241: DEMO: Enable shipping task log messages from anywhere

2023-07-14 Thread via GitHub
dstandish commented on code in PR #31241: URL: https://github.com/apache/airflow/pull/31241#discussion_r1264079820 ## airflow/utils/log/file_task_handler.py: ## @@ -218,9 +242,12 @@ def close(self): if self.handler: self.handler.close() -def _render_f

[GitHub] [airflow] jerell-mendoza opened a new issue, #32617: Issue with secrets_masker.py

2023-07-14 Thread via GitHub
jerell-mendoza opened a new issue, #32617: URL: https://github.com/apache/airflow/issues/32617 ### Apache Airflow version 2.6.3 ### What happened While running an ETL we are getting this warning in the logs `[2023-07-14, 19:12:32 UTC] {trigger_dagrun.py:197} INFO -

[GitHub] [airflow] dstandish commented on a diff in pull request #32616: Use `isdisjoint` instead of `not intersection`

2023-07-14 Thread via GitHub
dstandish commented on code in PR #32616: URL: https://github.com/apache/airflow/pull/32616#discussion_r1264071242 ## airflow/www/utils.py: ## @@ -926,6 +926,6 @@ def should_show(self, securitymanager: SecurityManager) -> bool: # Unable to obtain user role - de

[GitHub] [airflow] ferruzzi commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
ferruzzi commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636287115 I am almost certainly overthinking this but if we're considering Future Work... what about instead of setting a hardcoded time limit, we stored the previous durations somewhere and cal

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
ferruzzi commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1264033917 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-for

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32589: BUGFIX: AWS ECS trigger returned "success" after max_attempts instead of failing

2023-07-14 Thread via GitHub
vandonr-amz commented on code in PR #32589: URL: https://github.com/apache/airflow/pull/32589#discussion_r1264020600 ## airflow/providers/amazon/aws/triggers/ecs.py: ## @@ -179,8 +181,7 @@ async def run(self) -> AsyncIterator[TriggerEvent]: finally:

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-14 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1264017156 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -734,36 +733,72 @@ def __init__( *, db_identifier: str, db_type: RdsDbType | st

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-14 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1264015539 ## airflow/providers/amazon/aws/utils/rds.py: ## @@ -22,5 +22,5 @@ class RdsDbType(Enum): """Only available types for the RDS.""" -INSTANCE: str = "in

[airflow] branch main updated: Revert "Convert pull request template to YAML (#32597)" (#32615)

2023-07-14 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 42cb9693e4 Revert "Convert pull request template

[GitHub] [airflow] uranusjr merged pull request #32615: Revert "Convert pull request template to YAML (#32597)"

2023-07-14 Thread via GitHub
uranusjr merged PR #32615: URL: https://github.com/apache/airflow/pull/32615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] uranusjr opened a new pull request, #32615: Revert "Convert pull request template to YAML (#32597)"

2023-07-14 Thread via GitHub
uranusjr opened a new pull request, #32615: URL: https://github.com/apache/airflow/pull/32615 This reverts commit 0584d1342a9b74484b4e43eade850ad67caa7a23. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263965621 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-fo

[airflow] branch main updated: Speed up calculation of leaves and roots for task groups (#32592)

2023-07-14 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fad297c900 Speed up calculation of leaves and ro

[GitHub] [airflow] dstandish merged pull request #32592: Speed up calculation of leaves and roots for task groups

2023-07-14 Thread via GitHub
dstandish merged PR #32592: URL: https://github.com/apache/airflow/pull/32592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch constraints-main updated: Updating constraints. Github run id:5556276491

2023-07-14 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 0beb24bdd7 Updating con

[GitHub] [airflow] wolvery opened a new issue, #32614: Helm operator/sensor for Airflow

2023-07-14 Thread via GitHub
wolvery opened a new issue, #32614: URL: https://github.com/apache/airflow/issues/32614 ### Description A collection of Operators/Sensor to Helm seems something interesting to create in-house Helm Charts and use the Airflow to deploy these in Kubernetes. Some actions like Upgrade/Ins

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263965621 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-fo

[airflow-site] branch gh-pages updated (47c4a3067c -> 485502aeb9)

2023-07-14 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard 47c4a3067c Rewritten history to remove past gh-pages deployments new 485502aeb9 Rewritten history to

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-14 Thread via GitHub
amoghrajesh commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1263961144 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -805,10 +805,7 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo ```shell script

[GitHub] [airflow] potiuk commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263960362 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-forma

[GitHub] [airflow] potiuk commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263960362 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-forma

[GitHub] [airflow] pankajastro commented on pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
pankajastro commented on PR #32607: URL: https://github.com/apache/airflow/pull/32607#issuecomment-1636144965 > ah @pankajastro I saw yo ualso looked at this one :) yeah, actually because apache-hive provider (sasl )only. -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] potiuk commented on a diff in pull request #32613: Revising instructions to run back referencing scripts

2023-07-14 Thread via GitHub
potiuk commented on code in PR #32613: URL: https://github.com/apache/airflow/pull/32613#discussion_r1263956933 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -805,10 +805,7 @@ Documentation for providers can be found in the ``/docs/apache-airflow`` directo ```shell script

[GitHub] [airflow] o-nikolas commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636141927 > The tricky part is that our tests are executed in various "environments" - i.e. they can often be run in parallel with other tests in order to speed up the execution - and that migh

[GitHub] [airflow] amoghrajesh commented on pull request #32613: Revising instructions to run back referencing scripts

2023-07-14 Thread via GitHub
amoghrajesh commented on PR #32613: URL: https://github.com/apache/airflow/pull/32613#issuecomment-1636140121 @potiuk looks to me that this is the final PR to wind up the issue :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-14 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1263953725 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -734,36 +733,72 @@ def __init__( *, db_identifier: str, db_type: RdsDbType | str =

[GitHub] [airflow] amoghrajesh opened a new pull request, #32613: Revising instructions to run back referencing scripts

2023-07-14 Thread via GitHub
amoghrajesh opened a new pull request, #32613: URL: https://github.com/apache/airflow/pull/32613 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regar

[airflow-site] branch main updated: Purging post-docs to promote breeze publish-docs (#835)

2023-07-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new f22af86ee0 Purging post-docs to promote breeze

[GitHub] [airflow-site] potiuk merged pull request #835: Purging post-docs to promote breeze publish-docs

2023-07-14 Thread via GitHub
potiuk merged PR #835: URL: https://github.com/apache/airflow-site/pull/835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-14 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1263947482 ## airflow/providers/amazon/aws/utils/rds.py: ## @@ -22,5 +22,5 @@ class RdsDbType(Enum): """Only available types for the RDS.""" -INSTANCE: str = "insta

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263947337 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-fo

[GitHub] [airflow] potiuk commented on pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
potiuk commented on PR #32612: URL: https://github.com/apache/airflow/pull/32612#issuecomment-1636128933 Yeah. I think it's a good idea and we already have some of that in a different form - not necessarily timing but definitely targetted to balooning time of certain operations We h

[GitHub] [airflow-site] amoghrajesh commented on pull request #835: Purging post-docs to promote breeze publish-docs

2023-07-14 Thread via GitHub
amoghrajesh commented on PR #835: URL: https://github.com/apache/airflow-site/pull/835#issuecomment-1636128368 @potiuk the next step to remove post-docs from airflow-site is up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [airflow-site] amoghrajesh opened a new pull request, #835: Purging post-docs to promote breeze publish-docs

2023-07-14 Thread via GitHub
amoghrajesh opened a new pull request, #835: URL: https://github.com/apache/airflow-site/pull/835 As part of the issue: https://github.com/apache/airflow/issues/32491, the last PR: https://github.com/apache/airflow/pull/32594 added the post-docs task into a separate breeze command. W

[GitHub] [airflow] uranusjr commented on a diff in pull request #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
uranusjr commented on code in PR #32612: URL: https://github.com/apache/airflow/pull/32612#discussion_r1263945074 ## tests/cli/test_cli_parser.py: ## @@ -268,3 +269,13 @@ def test_invalid_choice_raises_for_export_format_in_db_export_archived_command( f"--export-for

[GitHub] [airflow] o-nikolas opened a new pull request, #32612: Add a test to catch regressions in cli run time

2023-07-14 Thread via GitHub
o-nikolas opened a new pull request, #32612: URL: https://github.com/apache/airflow/pull/32612 A lot of work has been done to optimize load time of the Executor modules so that they can be included in the CLI code path (so that executors can vend cli commands, part of AIP-51, see [this PR]

[GitHub] [airflow] potiuk commented on pull request #32594: Implement the breeze add-back-references command

2023-07-14 Thread via GitHub
potiuk commented on PR #32594: URL: https://github.com/apache/airflow/pull/32594#issuecomment-1636105687 Cool. Now you need to update release instructions and remove the old script from airlfow-sites as a follow up. -- This is an automated message from the Apache Git Service. To respond

[airflow] branch main updated: Implement the breeze add-back-references command (#32594)

2023-07-14 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 945f48a1fd Implement the breeze add-back-references

[GitHub] [airflow] potiuk merged pull request #32594: Implement the breeze add-back-references command

2023-07-14 Thread via GitHub
potiuk merged PR #32594: URL: https://github.com/apache/airflow/pull/32594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
potiuk commented on PR #32607: URL: https://github.com/apache/airflow/pull/32607#issuecomment-1636100857 Updated. Also removed the "sasl" direct inclusion. Let's see :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [airflow] o-nikolas commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-14 Thread via GitHub
o-nikolas commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1636099840 > I can make an example DAG, but I don't know how useful that would be at the moment. Once more operators are made, we can always edit the example DAG, though. @mahammi yeah I

[GitHub] [airflow] potiuk commented on pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
potiuk commented on PR #32607: URL: https://github.com/apache/airflow/pull/32607#issuecomment-1636098769 > It should be pyhive[hive_pure_sasl]>=0.7.1.dev0 Yeah. Seems like we have to combine the two :) -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-14 Thread via GitHub
o-nikolas commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1263921983 ## airflow/www/fab_security/manager.py: ## @@ -208,12 +206,6 @@ def oauth_tokengetter(token=None): userstatschartview = UserStatsChartView permissionmodel

[GitHub] [airflow] potiuk commented on issue #32600: Scheduler stop importing DAG after failed task

2023-07-14 Thread via GitHub
potiuk commented on issue #32600: URL: https://github.com/apache/airflow/issues/32600#issuecomment-1636086477 Yep. If there is memory or other reasons why resources are missing, this is fairly reasonable to expect that any system will be unstable and until you restart it there is no way it

[airflow] branch main updated: Support setting dependencies for tasks called outside TaskGroup context manager (#32351)

2023-07-14 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1a1753c724 Support setting dependencies fo

[GitHub] [airflow] ephraimbuddy merged pull request #32351: Support setting dependencies for tasks called outside TaskGroup context manager

2023-07-14 Thread via GitHub
ephraimbuddy merged PR #32351: URL: https://github.com/apache/airflow/pull/32351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[airflow] branch main updated: Convert pull request template to YAML (#32597)

2023-07-14 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0584d1342a Convert pull request template to YAML

[GitHub] [airflow] uranusjr merged pull request #32597: Convert pull request template to YAML

2023-07-14 Thread via GitHub
uranusjr merged PR #32597: URL: https://github.com/apache/airflow/pull/32597 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] mdeshmu commented on pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
mdeshmu commented on PR #32607: URL: https://github.com/apache/airflow/pull/32607#issuecomment-1636023342 It should be pyhive[hive_pure_sasl]>=0.7.1.dev0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] potiuk commented on pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
potiuk commented on PR #32607: URL: https://github.com/apache/airflow/pull/32607#issuecomment-1636011705 ah @pankajastro I saw yo ualso looked at this one :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [airflow] hussein-awala opened a new pull request, #32608: Add an option to use a direct DB connection in KEDA when pgbouncer is enabled

2023-07-14 Thread via GitHub
hussein-awala opened a new pull request, #32608: URL: https://github.com/apache/airflow/pull/32608 closes: #32590 This PR introduces a new conf `workers.keda.usePgbouncer` with a default value of true. This parameter is used to determine whether to utilize a pgbouncer con

[GitHub] [airflow] potiuk closed pull request #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
potiuk closed pull request #32607: Test 0.7.1dev0 version relased with 3.11 support URL: https://github.com/apache/airflow/pull/32607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [airflow] potiuk opened a new pull request, #32607: Test 0.7.1dev0 version relased with 3.11 support

2023-07-14 Thread via GitHub
potiuk opened a new pull request, #32607: URL: https://github.com/apache/airflow/pull/32607 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for m

[GitHub] [airflow] vincbeck commented on pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-14 Thread via GitHub
vincbeck commented on PR #28900: URL: https://github.com/apache/airflow/pull/28900#issuecomment-1635986869 > > Would be a good idea to change many of the functions to kwarg-only (adding a * in front of all arguments) > > I guess you refer to private functions I introduced such as `_s

  1   2   >