geido commented on code in PR #45029: URL: https://github.com/apache/superset/pull/45029#discussion_r4219693333
########## superset/config.py: ########## @@ -1897,6 +1900,11 @@ def _normalize_version_history_retention_days(value: object, *, legacy: bool) -> VERSION_HISTORY_RETENTION_DAYS: int = _parse_version_history_retention_days() +# Maximum committed transactions pruned per scheduled run; 0 or None is unlimited. +# Set an int in superset_config.py; this key has no automatic environment parsing. +VERSION_HISTORY_PRUNE_MAX_TRANSACTIONS_PER_RUN: int | None = 1000 Review Comment: Non-blocking: with both retention tasks on a daily beat, this default also becomes a steady-state ceiling of 1000 pruned transactions a day, not only a bound on a catch-up run. Capture is on by default and a chart, dashboard or dataset save usually writes a transaction, so an instance that saves more than that per day never drains. `cap_reached` stays at 1 and history keeps growing. `SOFT_DELETE_PURGE_MAX_PER_RUN` has the same math for archived objects. The docs say a large backlog drains over successive runs, which only holds while daily inflow stays below the cap. Worth adding a line to both guides: size the cap (or run the task more often) above daily inflow, and treat `cap_reached` staying at 1 across runs as falling behind. Since each 1000-candidate window is already its own bounded transaction, a higher default for version history might also be reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
