Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
o-nikolas commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-2000323563 Looks like the diagrams weren't updated along with this migration? When you run precommit locally now, diagram changes are made for code that is unrelated. -- This is an automat

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
pateash commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1999613179 > 🎉 🎉 🎉 Thanks a lot @eladkal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
eladkal commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1999508072 🎉 🎉 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
eladkal merged PR #36935: URL: https://github.com/apache/airflow/pull/36935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
eladkal commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1999401863 > @eladkal some myPy checks are failing due to issues in azure/cosmos provider > > ``` > airflow/providers/microsoft/azure/hooks/cosmos.py:228: error: Argument > "partition_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-15 Thread via GitHub
pateash commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1999156115 @eladkal some myPy checks are failing due to issues in azure/cosmos provider ``` airflow/providers/microsoft/azure/hooks/cosmos.py:228: error: Argument "partition_key" to "create

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1525362164 ## airflow/models/dag.py: ## @@ -364,6 +364,8 @@ class DAG(LoggingMixin): :param max_active_runs: maximum number of active DAG runs, beyond this number

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1998132823 Doc build now works The failing tests are because of breaking main branch due to https://github.com/apache/airflow/pull/38123 once resolving in main rebase should fix it and we shoul

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1525323732 ## airflow/models/dag.py: ## @@ -364,6 +364,8 @@ class DAG(LoggingMixin): :param max_active_runs: maximum number of active DAG runs, beyond this number

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
pateash commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1997526443 > We are still missing doc entry that explain this feature and explain this is experimental one. Simply adding a short paragraph about it `DAG pausing, deactivation and deletion` paragr

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1524699590 ## airflow/config_templates/config.yml: ## @@ -326,7 +337,7 @@ core: version_added: 2.3.0 type: integer example: ~ - default: "" + defa

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1524699590 ## airflow/config_templates/config.yml: ## @@ -326,7 +337,7 @@ core: version_added: 2.3.0 type: integer example: ~ - default: "" + defa

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1524699590 ## airflow/config_templates/config.yml: ## @@ -326,7 +337,7 @@ core: version_added: 2.3.0 type: integer example: ~ - default: "" + defa

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1524655160 ## airflow/config_templates/config.yml: ## @@ -326,7 +337,7 @@ core: version_added: 2.3.0 type: integer example: ~ - default: "" + defa

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1996874358 We are still missing doc entry that explain this feature and explain this is experimental one. Simply adding a short paragraph about it `DAG pausing, deactivation and deletion` parag

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-14 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1524445822 ## airflow/config_templates/config.yml: ## @@ -116,6 +116,17 @@ core: type: string example: ~ default: "16" +max_consecutive_failed_dag_runs_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-11 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1519701026 ## airflow/config_templates/config.yml: ## @@ -321,12 +332,12 @@ core: default_task_execution_timeout: description: | The default task execution_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-11 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1519697784 ## airflow/models/dagrun.py: ## @@ -787,6 +826,16 @@ def recalculate(self) -> _UnfinishedStates: msg="task_failure", ) +

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512561027 ## airflow/models/dagrun.py: ## @@ -787,6 +826,16 @@ def recalculate(self) -> _UnfinishedStates: msg="task_failure", ) +

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512560651 ## airflow/models/dagrun.py: ## @@ -787,6 +826,16 @@ def recalculate(self) -> _UnfinishedStates: msg="task_failure", ) +

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512559990 ## airflow/models/dagrun.py: ## @@ -561,6 +561,45 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512558868 ## airflow/models/dagrun.py: ## @@ -561,6 +561,45 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512557168 ## airflow/models/dagrun.py: ## @@ -561,6 +561,45 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512552006 ## airflow/models/dag.py: ## @@ -3922,6 +3942,7 @@ def dag( concurrency: int | None = None, max_active_tasks: int = airflow_conf.getint("core", "max_activ

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512551277 ## airflow/models/dag.py: ## @@ -3574,6 +3588,7 @@ class DagModel(Base): max_active_tasks = Column(Integer, nullable=False) max_active_runs = Column(Inte

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512550655 ## airflow/models/dag.py: ## @@ -615,6 +618,16 @@ def __init__( self.last_loaded: datetime = timezone.utcnow() self.safe_dag_id = dag_id.replace(".

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512550197 ## airflow/models/dag.py: ## @@ -454,6 +456,7 @@ def __init__( concurrency: int | None = None, max_active_tasks: int = airflow_conf.getint("core",

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-05 Thread via GitHub
uranusjr commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1512548059 ## airflow/config_templates/config.yml: ## @@ -321,12 +332,12 @@ core: default_task_execution_timeout: description: | The default task execution

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-03-04 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1511797496 ## airflow/config_templates/config.yml: ## @@ -116,6 +116,17 @@ core: type: string example: ~ default: "16" +max_consecutive_failed_dag_runs_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-23 Thread via GitHub
amoghrajesh commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1500770044 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_act

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-23 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1500672202 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-23 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1500672202 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-22 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1499095149 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-22 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1499091455 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-22 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1499091455 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-22 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1499091455 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-22 Thread via GitHub
eladkal commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1498975866 ## airflow/api_connexion/schemas/dag_schema.py: ## @@ -69,6 +69,7 @@ class Meta: tags = fields.List(fields.Nested(DagTagSchema), dump_only=True) max_active_

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-15 Thread via GitHub
potiuk commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1947220665 This time I rebased it, but you can usually do it yourself without calling maintainers -> generally I recommend to rebase eaarly, often, and especially when you see you are behind and se

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-15 Thread via GitHub
potiuk commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1947217035 > The Test failures look like are coming from other changes cc. @eladkal @potiuk Yes. Usually rebasing helps. Those errors were found and fixed earlier today -- This is an autom

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-15 Thread via GitHub
pateash commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1946849448 The failures look like are coming from other changes cc. @eladkal @potiuk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-12 Thread via GitHub
pateash commented on PR #36935: URL: https://github.com/apache/airflow/pull/36935#issuecomment-1940482538 ### TODO: - [ ] adding test for core config - [ ] adding test for environment variable -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-04 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1474223517 ## airflow/models/dagrun.py: ## @@ -498,6 +498,37 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars(

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-01 Thread via GitHub
pateash commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1474223517 ## airflow/models/dagrun.py: ## @@ -498,6 +498,37 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars(

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

2024-02-01 Thread via GitHub
dirrao commented on code in PR #36935: URL: https://github.com/apache/airflow/pull/36935#discussion_r1474086299 ## airflow/models/dagrun.py: ## @@ -498,6 +498,37 @@ def fetch_task_instances( tis = tis.where(TI.task_id.in_(task_ids)) return session.scalars(t