mikebridge commented on code in PR #45029: URL: https://github.com/apache/superset/pull/45029#discussion_r4221865873
########## UPDATING.md: ########## @@ -1732,6 +1734,8 @@ Soft-deleted dashboards, charts, and datasets are now permanently removed after Purging is **live by default** (`SOFT_DELETE_PURGE_DRY_RUN=False`), so the retention promise above is real on a stock deployment. Set it to `True` to have the task log `would_purge` counts and delete nothing — the lever is retained, so an operator can return to dry-run at any time. Note `would_purge` is an **upper bound** — it counts every entity past the retention window without evaluating deletion blockers, so a real run may purge fewer (entities referenced by report schedules or set as a user's welcome dashboard are blocked and reported separately). The task only acts while the `SOFT_DELETE` rollout flag is on; it now ships on by default. +**Scheduled-run cap (behavior change):** `SOFT_DELETE_PURGE_MAX_PER_RUN` defaults to 1000 successful root-entity purges across charts, dashboards, and datasets per invocation. One root plus its cascade counts as one; physical row counts are separate. The cap bounds committed deletions, not candidate evaluations: blocked roots are re-evaluated on every run, so a blocked-heavy backlog can still require substantial work. Set `0` or `None` for unlimited scheduled purging. Invalid values skip the task before deletion. Capped runs report `cap_reached` and `remaining_eligible` (aged supported roots, including blocked roots); `remaining_count_complete=False` and a null remainder signal a post-commit measurement failure without hiding successful purge totals. Model priority rotates by day so a sustained backlog in one model does not indefinitely exclude the others. Dry-run still counts the entire eligible backlog without writes and reports `eligible_backlog` and `estimated_capped_runs`; this is an upper-bound estimate when references block deletion. The cap is per task invocation, not a quota shared across concurrent workers. The manual `superset deletion-retention force-purge` command stays uncapped. Review Comment: Good catch, addressed in `d5cb98e100`. UPDATING and the archive guide now name the three cases (scan failure, uncertain commit and remainder-measurement failure, including runs with no confirmed purge) and point to `scan_failures`, `commit_uncertain`, the remainder-failure counter and the logs. ########## superset/config.py: ########## @@ -1897,6 +1900,11 @@ def _normalize_version_history_retention_days(value: object, *, legacy: bool) -> VERSION_HISTORY_RETENTION_DAYS: int = _parse_version_history_retention_days() +# Maximum committed transactions pruned per scheduled run; 0 or None is unlimited. +# Set an int in superset_config.py; this key has no automatic environment parsing. +VERSION_HISTORY_PRUNE_MAX_TRANSACTIONS_PER_RUN: int | None = 1000 Review Comment: Good point, addressed in `d5cb98e100`. Both guides now explain that at 1,000 units/day, 1,500 newly eligible units/day adds 500/day of backlog, and recommend sizing the cap or beat frequency above eligible inflow with catch-up headroom. Could we keep the defaults as they are here, and let repeated cap hits plus the backlog/completeness and failure signals guide operator tuning? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
