[
https://issues.apache.org/jira/browse/AIRFLOW-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281634#comment-15281634
]
Chris Riccomini commented on AIRFLOW-108:
-----------------------------------------
I haven't fully thought through how this would be implemented, but we have 100s
of DAGs that run every 15m. The DB is starting to get larger and larger. The DB
size isn't a problem now, but I'm worried that it might be shortly.
> Add data retention policy to Airflow
> ------------------------------------
>
> Key: AIRFLOW-108
> URL: https://issues.apache.org/jira/browse/AIRFLOW-108
> Project: Apache Airflow
> Issue Type: Wish
> Components: db, scheduler
> Reporter: Chris Riccomini
>
> Airflow's DB currently holds the entire history of all executions for all
> time. This is problematic as the DB grows. The UI starts to get slower, and
> the DB's disk usage grows. There is no bound to how large the DB will grow.
> It would be useful to add a feature in Airflow to do two things:
> # Delete old data from the DB
> # Mark some lower watermark, past which DAG executions are ignored
> For example, (2) would allow you to tell the scheduler "ignore all data prior
> to a year ago". And (1) would allow Airflow to delete all data prior to
> January 1, 2015.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
