[
https://issues.apache.org/jira/browse/GOBBLIN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Varshney updated GOBBLIN-2026:
------------------------------------
Description:
Currently, while cleaning the log files, the Retention job goes into OOM and
silently fails when the no of log files is too many.
As the job silently fails, user doesn't get to know explicitly about it. Hence,
when going into OOM, retention job should explicitly fail if it can't be
proceeded further
was:
Currently, while cleaning the log files, the Retention job goes into OOM and
fails when the no of log files is too many.
Retention job while fetching all the dataset versions, loads the file status
all at once into the memory, resulting in this issue.
Thus, the Retention job should avoid loading all data into memory, and use an
iterator-based approach. This will load only limited file status into memory
and making the retention job pipeline more robust to OOM errors
> CLONE - Retention Job should fail on OOM
> ----------------------------------------
>
> Key: GOBBLIN-2026
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2026
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: misc
> Reporter: Arpit Varshney
> Priority: Major
>
> Currently, while cleaning the log files, the Retention job goes into OOM and
> silently fails when the no of log files is too many.
> As the job silently fails, user doesn't get to know explicitly about it.
> Hence, when going into OOM, retention job should explicitly fail if it can't
> be proceeded further
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
