Lee-W commented on code in PR #66463:
URL: https://github.com/apache/airflow/pull/66463#discussion_r3195564690
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3025,6 +3014,29 @@ state_store:
type: string
example: "mypackage.state.CustomStateBackend"
default: "airflow.state.metastore.MetastoreStateBackend"
+ default_retention_days:
+ description: |
+ Number of days to retain task state rows after their last update.
Review Comment:
will it also affect asset state?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3047,6 +3054,19 @@ def _remove_unreferenced_triggers(self, *, session:
Session = NEW_SESSION) -> No
.execution_options(synchronize_session="fetch")
)
+ def _cleanup_expired_task_state(self) -> None:
+ """
+ Remove expired task state rows and orphaned asset state rows.
+
+ Reads ``[state_store] default_retention_days`` from config and
delegates to the
+ configured state backend. Runs on the interval set by ``[state_store]
state_cleanup_interval``.
+ """
+ retention_days = conf.getint("state_store", "default_retention_days")
Review Comment:
if `default_retention_days` is set to 0, then we'll delete everything older
than now, it doesn't look right 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
