[GitHub] [airflow] amoghrajesh commented on issue #32491: Implement `breeze publish-docs` command

2023-07-13 Thread via GitHub
amoghrajesh commented on issue #32491: URL: https://github.com/apache/airflow/issues/32491#issuecomment-1635372326 @potiuk should we close this issue and open a new one to track the `add-back-references`? I think that would be more convenient. -- This is an automated message from the Apa

[GitHub] [airflow] uranusjr commented on a diff in pull request #32592: Speed up calculation of leaves and roots for task groups

2023-07-13 Thread via GitHub
uranusjr commented on code in PR #32592: URL: https://github.com/apache/airflow/pull/32592#discussion_r1263355739 ## airflow/utils/task_group.py: ## @@ -357,31 +357,35 @@ def get_roots(self) -> Generator[BaseOperator, None, None]: Returns a generator of tasks that are

[GitHub] [airflow] amoghrajesh opened a new pull request, #32593: Fixing airflow_site_dir type

2023-07-13 Thread via GitHub
amoghrajesh opened a new pull request, #32593: URL: https://github.com/apache/airflow/pull/32593 Realised that the airflow_site_dir has a type bug and it was not caught so far. Changing it to a string. --- **^ Add meaningful description above** Read the **[Pull Reque

[GitHub] [airflow] eladkal commented on pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
eladkal commented on PR #32573: URL: https://github.com/apache/airflow/pull/32573#issuecomment-1635239279 > I think we have 2 options, > > 1. "fix" that code, but this changes the behavior for every usage, not sure of the consequences > 2. close this PR and keep our own helper func

[GitHub] [airflow] ferruzzi commented on pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
ferruzzi commented on PR #32573: URL: https://github.com/apache/airflow/pull/32573#issuecomment-1635173543 why_not_both.gif I'd say the code should be fixed to match the docs in that case, but also I get that that isn't what you signed up for here so I guess drop this for now and

[GitHub] [airflow] mis98zb commented on a diff in pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-13 Thread via GitHub
mis98zb commented on code in PR #32122: URL: https://github.com/apache/airflow/pull/32122#discussion_r1262265208 ## airflow/models/taskinstance.py: ## @@ -2580,6 +2580,31 @@ def get_num_running_task_instances(self, session: Session, same_dagrun=False) -> )

[GitHub] [airflow] ferruzzi commented on pull request #30496: [OTel Integration] Add tagging to existing stats

2023-07-13 Thread via GitHub
ferruzzi commented on PR #30496: URL: https://github.com/apache/airflow/pull/30496#issuecomment-1635171569 @conorbev "plans" may be a strong word, but I do intend to do more work on the OTel stuff as I get time. The reason they still have the embedded values is because not all metrics

[GitHub] [airflow] dstandish commented on pull request #32169: Add initial docs for setup / teardown

2023-07-13 Thread via GitHub
dstandish commented on PR #32169: URL: https://github.com/apache/airflow/pull/32169#issuecomment-1635108702 can I get an amen here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[airflow] branch main updated: Fix short circuit operator re teardowns (#32538)

2023-07-13 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new bb5307850d Fix short circuit operator re teardow

[GitHub] [airflow] dstandish merged pull request #32538: Fix short circuit operator re teardowns

2023-07-13 Thread via GitHub
dstandish merged PR #32538: URL: https://github.com/apache/airflow/pull/32538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] github-actions[bot] closed issue #31723: The Airflow scheduler crashes when configuring start_date for future date, setting schedule_interval and max_active_runs

2023-07-13 Thread via GitHub
github-actions[bot] closed issue #31723: The Airflow scheduler crashes when configuring start_date for future date, setting schedule_interval and max_active_runs URL: https://github.com/apache/airflow/issues/31723 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] dstandish commented on a diff in pull request #32538: Fix short circuit operator re teardowns

2023-07-13 Thread via GitHub
dstandish commented on code in PR #32538: URL: https://github.com/apache/airflow/pull/32538#discussion_r1263154303 ## airflow/operators/python.py: ## @@ -251,27 +253,37 @@ def execute(self, context: Context) -> Any: self.log.info("Proceeding with downstream tasks...

[GitHub] [airflow] bbovenzi commented on pull request #30478: Mark Task Groups as Success/Failure

2023-07-13 Thread via GitHub
bbovenzi commented on PR #30478: URL: https://github.com/apache/airflow/pull/30478#issuecomment-1635065722 > Will this change still work now that the task group button actions have been moved in the ui? Yep! -- This is an automated message from the Apache Git Service. To respond to

[airflow] branch main updated: Fix rendering of dagRunTimeout (#32565)

2023-07-13 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f1fc6dc4b9 Fix rendering of dagRunTimeout (#32565

[GitHub] [airflow] bbovenzi closed issue #32442: Dag run timeout with timedelta value is rendered as [object object] in UI

2023-07-13 Thread via GitHub
bbovenzi closed issue #32442: Dag run timeout with timedelta value is rendered as [object object] in UI URL: https://github.com/apache/airflow/issues/32442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] bbovenzi merged pull request #32565: Fix rendering of dagRunTimeout

2023-07-13 Thread via GitHub
bbovenzi merged PR #32565: URL: https://github.com/apache/airflow/pull/32565 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] hussein-awala commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-13 Thread via GitHub
hussein-awala commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1635049044 I asked to check if you set the group id. > I tried to set consumers count less, equal and more than partitions but every time the error happened. When you tried set

[GitHub] [airflow] bbovenzi commented on pull request #32578: Add setup/teardown info to grid/graph view

2023-07-13 Thread via GitHub
bbovenzi commented on PR #32578: URL: https://github.com/apache/airflow/pull/32578#issuecomment-1635047887 Oops. I was too eager to merge. If anyone has any comments, I'm happy to fix them in a follow-up before we release 2.7.0 -- This is an automated message from the Apache Git Servi

[GitHub] [airflow] ikholodkov commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-13 Thread via GitHub
ikholodkov commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1635041727 @hussein-awala something like this { "bootstrap.servers": "server1:9093,server2:9093", "group.id": "test-consumer-group", "auto.offset.reset": "latest", "ss

[GitHub] [airflow] hussein-awala commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-13 Thread via GitHub
hussein-awala commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1635036781 what do you have in the extra of your connection? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[airflow] branch main updated: Fix dags list counts (#32579)

2023-07-13 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 41164dd663 Fix dags list counts (#32579) 41164dd

[GitHub] [airflow] kaxil merged pull request #32579: Fix dags list counts

2023-07-13 Thread via GitHub
kaxil merged PR #32579: URL: https://github.com/apache/airflow/pull/32579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.ap

[GitHub] [airflow] kaxil commented on pull request #32579: Fix dags list counts

2023-07-13 Thread via GitHub
kaxil commented on PR #32579: URL: https://github.com/apache/airflow/pull/32579#issuecomment-1634994097 I am going to merge this as I know @bbovenzi is OOO. @phanikumv can you add that test in a separate PR, please? -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [airflow] conorbev commented on pull request #30496: [OTel Integration] Add tagging to existing stats

2023-07-13 Thread via GitHub
conorbev commented on PR #30496: URL: https://github.com/apache/airflow/pull/30496#issuecomment-1634981952 hey @ferruzzi I'm curious about some of these metrics and if you had further plans to change them. Like e.g.: `Stats.incr(f"ti.start.{self.task.dag_id}.{self.task.task_id}", tags=self

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32563: Add Deferrable mode to StepFunctionStartExecutionOperator

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32563: URL: https://github.com/apache/airflow/pull/32563#discussion_r1263094816 ## airflow/providers/amazon/aws/operators/step_function.py: ## @@ -74,9 +88,25 @@ def execute(self, context: Context): raise AirflowException(f"Faile

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32563: Add Deferrable mode to StepFunctionStartExecutionOperator

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32563: URL: https://github.com/apache/airflow/pull/32563#discussion_r1263094816 ## airflow/providers/amazon/aws/operators/step_function.py: ## @@ -74,9 +88,25 @@ def execute(self, context: Context): raise AirflowException(f"Faile

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1263090829 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -733,22 +729,42 @@ def __init__( *, db_identifier: str, db_type: RdsDbType | st

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32572: Reduce config's default for max TIs per query & enforce <= parallelism

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32572: URL: https://github.com/apache/airflow/pull/32572#discussion_r1263075204 ## airflow/configuration.py: ## @@ -378,6 +379,27 @@ def validate(self): self._upgrade_postgres_metastore_conn() self.is_validated = True +

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
o-nikolas commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1263072460 ## airflow/www/fab_security/manager.py: ## @@ -208,12 +206,6 @@ def oauth_tokengetter(token=None): userstatschartview = UserStatsChartView permissionmodel

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32591: D205 Support - Utils

2023-07-13 Thread via GitHub
ferruzzi commented on code in PR #32591: URL: https://github.com/apache/airflow/pull/32591#discussion_r1263069030 ## airflow/utils/log/logging_mixin.py: ## @@ -174,15 +172,18 @@ def flush(self): def isatty(self): """ Returns False to indicate the fd is not

[GitHub] [airflow] bbovenzi merged pull request #32578: Add setup/teardown info to grid/graph view

2023-07-13 Thread via GitHub
bbovenzi merged PR #32578: URL: https://github.com/apache/airflow/pull/32578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated (44b4a3755d -> a58dcbb41d)

2023-07-13 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 44b4a3755d Add deferrable mode to `BeamRunPythonPipelineOperator` (#31471) add a58dcbb41d Add setup/teardown in

[GitHub] [airflow] vandonr-amz commented on pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
vandonr-amz commented on PR #32573: URL: https://github.com/apache/airflow/pull/32573#issuecomment-1634943125 I think we have 2 options, 1. "fix" that code, but this changes the behavior for every usage, not sure of the consequences 2. close this PR and keep our own helper function

[GitHub] [airflow] vandonr-amz commented on pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
vandonr-amz commented on PR #32573: URL: https://github.com/apache/airflow/pull/32573#issuecomment-1634934797 > > Interesting, I wonder if those failing Lambda tests were written wrong or if these two functions are somehow not actually functionally identical. > > Yeah same, I'll take

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1263015839 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -105,10 +115,9 @@ def __init__( db_snapshot_identifier: str, tags: Sequence[TagTypeDef]

[GitHub] [airflow] jens-scheffler-bosch commented on issue #32586: Logging Details Message to Address Scheduler Heartbeat Loss and Downtime Challenges

2023-07-13 Thread via GitHub
jens-scheffler-bosch commented on issue #32586: URL: https://github.com/apache/airflow/issues/32586#issuecomment-1634841750 Hi @BShraman thanks for filing this issue. I marked it as improvement as it something that enhances the situation. I understand that users expect a very reliable

[GitHub] [airflow] renzepost commented on a diff in pull request #32465: Fix truncated execution_date in task instance details

2023-07-13 Thread via GitHub
renzepost commented on code in PR #32465: URL: https://github.com/apache/airflow/pull/32465#discussion_r1262977293 ## airflow/www/static/js/task.js: ## @@ -25,12 +25,17 @@ import validator from "validator"; document.addEventListener("DOMContentLoaded", () => { document.query

[GitHub] [airflow] mahammi commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-13 Thread via GitHub
mahammi commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1634716859 > > The failing test has a suggestion to add the operator to a "no example available" list [here](https://github.com/apache/airflow/blob/main/tests/always/test_project_structure.py#L284

[GitHub] [airflow] vincbeck commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1262922283 ## airflow/www/fab_security/manager.py: ## @@ -208,12 +206,6 @@ def oauth_tokengetter(token=None): userstatschartview = UserStatsChartView permissionmodelv

[GitHub] [airflow] boring-cyborg[bot] commented on issue #32590: When using KEDA and pgbouncer together, KEDA logs repeated prepared statement errors

2023-07-13 Thread via GitHub
boring-cyborg[bot] commented on issue #32590: URL: https://github.com/apache/airflow/issues/32590#issuecomment-1634703954 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] brokenjacobs opened a new issue, #32590: When using KEDA and pgbouncer together, KEDA logs repeated prepared statement errors

2023-07-13 Thread via GitHub
brokenjacobs opened a new issue, #32590: URL: https://github.com/apache/airflow/issues/32590 ### Official Helm Chart version 1.10.0 (latest released) ### Apache Airflow version 2.6.2 ### Kubernetes Version v1.26.5-gke.1200 ### Helm Chart configuration

[GitHub] [airflow] syedahsn commented on a diff in pull request #32563: Add Deferrable mode to StepFunctionStartExecutionOperator

2023-07-13 Thread via GitHub
syedahsn commented on code in PR #32563: URL: https://github.com/apache/airflow/pull/32563#discussion_r1262918852 ## airflow/providers/amazon/aws/operators/step_function.py: ## @@ -74,9 +88,25 @@ def execute(self, context: Context): raise AirflowException(f"Failed t

[GitHub] [airflow] vincbeck commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1262916566 ## airflow/auth/managers/fab/security_manager_override.py: ## @@ -0,0 +1,218 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribut

[GitHub] [airflow] vincbeck commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1262914792 ## airflow/auth/managers/fab/security_manager_override.py: ## @@ -0,0 +1,218 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribut

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1262912492 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -733,22 +729,42 @@ def __init__( *, db_identifier: str, db_type: RdsDbType | str =

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1262912492 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -733,22 +729,42 @@ def __init__( *, db_identifier: str, db_type: RdsDbType | str =

[GitHub] [airflow] dstandish commented on a diff in pull request #32538: Fix short circuit operator re teardowns

2023-07-13 Thread via GitHub
dstandish commented on code in PR #32538: URL: https://github.com/apache/airflow/pull/32538#discussion_r1262910810 ## airflow/operators/python.py: ## @@ -251,27 +253,37 @@ def execute(self, context: Context) -> Any: self.log.info("Proceeding with downstream tasks...

[GitHub] [airflow] o-nikolas commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-13 Thread via GitHub
o-nikolas commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1634687942 > The failing test has a suggestion to add the operator to a "no example available" list [here](https://github.com/apache/airflow/blob/main/tests/always/test_project_structure.py#L284

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1262909549 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -105,10 +115,9 @@ def __init__( db_snapshot_identifier: str, tags: Sequence[TagTypeDef] | d

[GitHub] [airflow] syedahsn commented on a diff in pull request #32437: add deferrable mode to rds start & stop DB

2023-07-13 Thread via GitHub
syedahsn commented on code in PR #32437: URL: https://github.com/apache/airflow/pull/32437#discussion_r1262909549 ## airflow/providers/amazon/aws/operators/rds.py: ## @@ -105,10 +115,9 @@ def __init__( db_snapshot_identifier: str, tags: Sequence[TagTypeDef] | d

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
o-nikolas commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1262901259 ## airflow/configuration.py: ## @@ -1743,24 +1742,6 @@ def initialize_secrets_backends() -> list[BaseSecretsBackend]: return backend_list -def initialize_

[GitHub] [airflow] ferruzzi commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-13 Thread via GitHub
ferruzzi commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1634684246 I wrote a message about this last night, I wonder what happened to it. strange. The failing test has a suggestion to add the operator to a "no example available" list [here

[GitHub] [airflow] vandonr-amz commented on pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-13 Thread via GitHub
vandonr-amz commented on PR #32498: URL: https://github.com/apache/airflow/pull/32498#issuecomment-1634675854 > You're missing an example dag/system test for this new operator Should we create a "broken dag" that just shows how to use the operator for now, and focus and making it a fu

[GitHub] [airflow] NilsJPWerner commented on pull request #30478: Mark Task Groups as Success/Failure

2023-07-13 Thread via GitHub
NilsJPWerner commented on PR #30478: URL: https://github.com/apache/airflow/pull/30478#issuecomment-1634675767 Will this change still work now that the task group button actions have been moved in the ui? -- This is an automated message from the Apache Git Service. To respond to the messa

[airflow] branch constraints-main updated: Updating constraints. Github run id:5545640806

2023-07-13 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new d1827fb4b0 Updating con

[GitHub] [airflow] potiuk closed issue #32588: Documentation Update to stop breeze environment

2023-07-13 Thread via GitHub
potiuk closed issue #32588: Documentation Update to stop breeze environment URL: https://github.com/apache/airflow/issues/32588 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] darshilparmar opened a new issue, #32588: Documentation Update to stop breeze environment

2023-07-13 Thread via GitHub
darshilparmar opened a new issue, #32588: URL: https://github.com/apache/airflow/issues/32588 ### What do you see as an issue? Hello, This is the first time I am contributing to open source I was trying to set up Airflow Dev Environment on my local computer (MacOS M1) and I found t

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32572: Reduce config's default for max TIs per query & enforce <= parallelism

2023-07-13 Thread via GitHub
o-nikolas commented on code in PR #32572: URL: https://github.com/apache/airflow/pull/32572#discussion_r1262829467 ## airflow/config_templates/config.yml: ## @@ -2451,14 +2451,15 @@ scheduler: max_tis_per_query: description: | This changes the batch size of

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32572: Reduce config's default for max TIs per query & enforce <= parallelism

2023-07-13 Thread via GitHub
o-nikolas commented on code in PR #32572: URL: https://github.com/apache/airflow/pull/32572#discussion_r1262827347 ## airflow/configuration.py: ## @@ -378,6 +379,27 @@ def validate(self): self._upgrade_postgres_metastore_conn() self.is_validated = True +d

[GitHub] [airflow] vandonr-amz commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`

2023-07-13 Thread via GitHub
vandonr-amz commented on issue #32580: URL: https://github.com/apache/airflow/issues/32580#issuecomment-1634580545 Hmm that's interesting because `execution_timeout` is a property of the base operator (i.e. of all operators), which is usually handled "automatically", but I don't think that

[GitHub] [airflow] o-nikolas commented on a diff in pull request #32498: Add Eventbridge PutEvents operator and hook

2023-07-13 Thread via GitHub
o-nikolas commented on code in PR #32498: URL: https://github.com/apache/airflow/pull/32498#discussion_r1262818421 ## docs/apache-airflow-providers-amazon/operators/events.rst: ## @@ -0,0 +1,47 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contr

[airflow] branch main updated: Add deferrable mode to `BeamRunPythonPipelineOperator` (#31471)

2023-07-13 Thread pankaj
This is an automated email from the ASF dual-hosted git repository. pankaj pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 44b4a3755d Add deferrable mode to `BeamRunPythonPip

[GitHub] [airflow] pankajastro merged pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
pankajastro merged PR #31471: URL: https://github.com/apache/airflow/pull/31471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

[GitHub] [airflow] vandonr-amz commented on pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
vandonr-amz commented on PR #32573: URL: https://github.com/apache/airflow/pull/32573#issuecomment-1634546783 > Interesting, I wonder if those failing Lambda tests were written wrong or if these two functions are somehow not actually functionally identical. Yeah same, I'll take a look

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
vandonr-amz commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1262801825 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -21,13 +21,17 @@ from datetime import datetime from enum import Enum +from deprecated import deprecat

[GitHub] [airflow] jedcunningham commented on pull request #32578: Add setup/teardown info to grid/graph view

2023-07-13 Thread via GitHub
jedcunningham commented on PR #32578: URL: https://github.com/apache/airflow/pull/32578#issuecomment-1634520561 I've refactored the edge marking code a bit (this also fixed the tests), please take another look reviewers. -- This is an automated message from the Apache Git Service. To resp

[airflow] branch constraints-main updated: Updating constraints. Github run id:5544357735

2023-07-13 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new f6fccdc73b Updating con

[GitHub] [airflow] VladaZakharova commented on pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
VladaZakharova commented on PR #31471: URL: https://github.com/apache/airflow/pull/31471#issuecomment-1634485196 @pankajastro I have fixed this small mistake, could you please check? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [airflow] VladaZakharova commented on a diff in pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
VladaZakharova commented on code in PR #31471: URL: https://github.com/apache/airflow/pull/31471#discussion_r1262750959 ## tests/system/providers/apache/beam/example_python_async.py: ## @@ -0,0 +1,137 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1262749790 ## airflow/models/dag.py: ## @@ -1378,6 +1379,37 @@ def normalized_schedule_interval(self) -> ScheduleInterval: _schedule_interval = self.schedule_inte

[GitHub] [airflow] nehemiascr commented on issue #32375: sqlalchemy.exc.ResourceClosedError when updating a task instance's executor_config from a task_instance_mutation_hook

2023-07-13 Thread via GitHub
nehemiascr commented on issue #32375: URL: https://github.com/apache/airflow/issues/32375#issuecomment-1634466429 Jens, I added `finally:` statements to the code and the issue still persists, to answer your question, yes, it is a different MySQL database, airflow is running on its own Postg

[GitHub] [airflow] vincbeck commented on a diff in pull request #32573: remove provider specific helper in favor of equivalent one in core

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #32573: URL: https://github.com/apache/airflow/pull/32573#discussion_r1262699596 ## airflow/providers/amazon/aws/utils/__init__.py: ## @@ -21,13 +21,17 @@ from datetime import datetime from enum import Enum +from deprecated import deprecated

[GitHub] [airflow] vincbeck commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1262690569 ## airflow/www/extensions/init_auth_manager.py: ## @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agre

[GitHub] [airflow] rohan472000 commented on issue #31826: Add a command line option to start a triggerer process with breeze

2023-07-13 Thread via GitHub
rohan472000 commented on issue #31826: URL: https://github.com/apache/airflow/issues/31826#issuecomment-1634400190 > @rohan472000 are you still working on this? yes..but not able to solve it yet. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1262679496 ## airflow/models/baseoperator.py: ## @@ -907,7 +907,7 @@ def __init__( max_active_tis_per_dag = task_concurrency self.max_active_tis_per_dag:

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-13 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1262678233 ## airflow/models/dag.py: ## @@ -1378,6 +1379,37 @@ def normalized_schedule_interval(self) -> ScheduleInterval: _schedule_interval = self.schedule_inte

[GitHub] [airflow] boring-cyborg[bot] commented on issue #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-13 Thread via GitHub
boring-cyborg[bot] commented on issue #32585: URL: https://github.com/apache/airflow/issues/32585#issuecomment-1634380099 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] ikholodkov opened a new issue, #32585: Commit failed: Local: No offset stored while using AwaitMessageTriggerFunctionSensor

2023-07-13 Thread via GitHub
ikholodkov opened a new issue, #32585: URL: https://github.com/apache/airflow/issues/32585 ### Apache Airflow version 2.6.3 ### What happened While trying to use AwaitMessageTriggerFunctionSensor i'm increasing count of dagruns. I've encountered an exception `cimpl.K

[airflow] branch main updated: Add deprecation info to the providers modules and classes docstring (#32536)

2023-07-13 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e01323635a Add deprecation info to the providers mo

[GitHub] [airflow] potiuk merged pull request #32536: Add deprecation info to the providers modules and classes docstring

2023-07-13 Thread via GitHub
potiuk merged PR #32536: URL: https://github.com/apache/airflow/pull/32536 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] pankajastro commented on a diff in pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
pankajastro commented on code in PR #31471: URL: https://github.com/apache/airflow/pull/31471#discussion_r1262616087 ## tests/system/providers/apache/beam/example_python_async.py: ## @@ -0,0 +1,137 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

[GitHub] [airflow] pankajastro commented on pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
pankajastro commented on PR #31471: URL: https://github.com/apache/airflow/pull/31471#issuecomment-1634334017 > @potiuk @pankajastro > Hi! Do we have anything else to fix here? Hi just posted one small question otherwise looks good, -- This is an automated message from the Apach

[GitHub] [airflow] ikholodkov commented on issue #31795: AwaitMessageTriggerFunctionSensor not firing all eligble messages

2023-07-13 Thread via GitHub
ikholodkov commented on issue #31795: URL: https://github.com/apache/airflow/issues/31795#issuecomment-1634317356 @hussein-awala are you talking about the situation when a sensor could commit message twice? I mean, is it linked to [confluent-kafka-python](https://github.com/confluentinc/con

[GitHub] [airflow] phanikumv commented on pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-07-13 Thread via GitHub
phanikumv commented on PR #31925: URL: https://github.com/apache/airflow/pull/31925#issuecomment-1634300022 Rebase your PR please @MaksYermak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] VladaZakharova commented on pull request #31471: Add deferrable mode to `BeamRunPythonPipelineOperator`

2023-07-13 Thread via GitHub
VladaZakharova commented on PR #31471: URL: https://github.com/apache/airflow/pull/31471#issuecomment-1634291415 @potiuk @pankajastro Hi! Do we have anything else to fix here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] VladaZakharova opened a new pull request, #32584: Fix DataflowStartFlexTemplateOperator system test

2023-07-13 Thread via GitHub
VladaZakharova opened a new pull request, #32584: URL: https://github.com/apache/airflow/pull/32584 This PR fixes system test for DataflowStartFlexTemplateOperator --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/

[GitHub] [airflow] nathadfield commented on pull request #32582: Allow sc templating in logs PVC

2023-07-13 Thread via GitHub
nathadfield commented on PR #32582: URL: https://github.com/apache/airflow/pull/32582#issuecomment-1634181092 Closes #32581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] pankajkoti commented on pull request #32526: Move CeleryExecutor to the celery provider

2023-07-13 Thread via GitHub
pankajkoti commented on PR #32526: URL: https://github.com/apache/airflow/pull/32526#issuecomment-1634168684 > Release manager decides which providers to release. While providers are released from main (all or nothing, no cherry picking of commits) release manager can still decide to skip s

[GitHub] [airflow] Natalique commented on issue #32581: Add Templating for logs PVC storageClassName

2023-07-13 Thread via GitHub
Natalique commented on issue #32581: URL: https://github.com/apache/airflow/issues/32581#issuecomment-1634166327 @nathadfield Yes, I just submitted this one - https://github.com/apache/airflow/pull/32582 -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32582: Allow sc templating in logs PVC

2023-07-13 Thread via GitHub
boring-cyborg[bot] commented on PR #32582: URL: https://github.com/apache/airflow/pull/32582#issuecomment-1634165553 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] Natalique opened a new pull request, #32582: Allow sc templating in logs PVC

2023-07-13 Thread via GitHub
Natalique opened a new pull request, #32582: URL: https://github.com/apache/airflow/pull/32582 Add tpl function so that storageClassName can be templated when using persistent storage for logs. -- This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [airflow] Natalique opened a new issue, #32581: Add Templating for logs PVC storageClassName

2023-07-13 Thread via GitHub
Natalique opened a new issue, #32581: URL: https://github.com/apache/airflow/issues/32581 ### Description I'm using EFS For different envs in EKS and I have a few different storage class I need to pass airflow a templated storageClassName Implementation Already tested loc

[GitHub] [airflow] eladkal commented on issue #30883: BranchPythonOperator skips downstream tasks for all mapped instances in TaskGroup mapping

2023-07-13 Thread via GitHub
eladkal commented on issue #30883: URL: https://github.com/apache/airflow/issues/30883#issuecomment-1634118540 The milestone issues are not something that we actually use.. I will mark this one for 2.6.4 -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] potiuk commented on pull request #32514: build(pre-commit): add list of supported deferrable operators to doc

2023-07-13 Thread via GitHub
potiuk commented on PR #32514: URL: https://github.com/apache/airflow/pull/32514#issuecomment-1634097338 @eladkal ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[GitHub] [airflow] VladaZakharova commented on pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-07-13 Thread via GitHub
VladaZakharova commented on PR #31925: URL: https://github.com/apache/airflow/pull/31925#issuecomment-1634093386 Hi @potiuk @Lee-W Could you please check those changes if there is something to fix here? Thanks! -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] matthieucx commented on issue #30883: BranchPythonOperator skips downstream tasks for all mapped instances in TaskGroup mapping

2023-07-13 Thread via GitHub
matthieucx commented on issue #30883: URL: https://github.com/apache/airflow/issues/30883#issuecomment-1634092949 Hey @eladkal, thanks for merging the fix. The fix is part of milestone 2.6.3 but it does not seem it was included in that release. Is that normal ? -- This is an automated mes

[GitHub] [airflow] potiuk commented on issue #31054: Can not get more logs of DAGS on UI when deploy on Kubernetes

2023-07-13 Thread via GitHub
potiuk commented on issue #31054: URL: https://github.com/apache/airflow/issues/31054#issuecomment-1634090477 can you please provide more logging information and describe exactly the circumstances you have @chuxiangfeng ? That would help to focus the problem. -- This is an automated mess

[GitHub] [airflow] chuxiangfeng opened a new issue, #31054: Can not get more logs of DAGS on UI when deploy on Kubernetes

2023-07-13 Thread via GitHub
chuxiangfeng opened a new issue, #31054: URL: https://github.com/apache/airflow/issues/31054 ### Apache Airflow version 2.6.0 ### What happened After update airflow to 2.6.0 and apache-airflow-providers-cncf-kubernetes==6.1.0,Can not get more logs of DAGS on UI when dep

[GitHub] [airflow] mis98zb commented on a diff in pull request #32122: control task group concurrency: use allow list for mapping index

2023-07-13 Thread via GitHub
mis98zb commented on code in PR #32122: URL: https://github.com/apache/airflow/pull/32122#discussion_r1262423525 ## airflow/jobs/scheduler_job_runner.py: ## @@ -277,9 +284,44 @@ def __get_concurrency_maps(self, states: Iterable[TaskInstanceState], session: S .where

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #32351: Support setting dependencies for tasks called outside TaskGroup context manager

2023-07-13 Thread via GitHub
ephraimbuddy commented on code in PR #32351: URL: https://github.com/apache/airflow/pull/32351#discussion_r1262371712 ## airflow/models/taskmixin.py: ## @@ -165,6 +182,11 @@ class DAGNode(DependencyMixin, metaclass=ABCMeta): def node_id(self) -> str: raise NotImple

  1   2   >