[
https://issues.apache.org/jira/browse/FINERACT-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Saghy resolved FINERACT-2926.
----------------------------------
Resolution: Fixed
> StuckJobListener waits forever when a COB partition was killed
> --------------------------------------------------------------
>
> Key: FINERACT-2926
> URL: https://issues.apache.org/jira/browse/FINERACT-2926
> Project: Apache Fineract
> Issue Type: Bug
> Reporter: Arnold Galovics
> Assignee: Arnold Galovics
> Priority: Major
> Fix For: 1.16.0
>
>
> 🐛 Steps to Reproduce
> 1. Run Fineract 1.15.0 with batch manager and batch worker enabled in one
> container, business date enabled, and 2 loans lagging behind the COB date.
> 2. Start Loan COB with POST /jobs/\{id}?command=executeJob.
> 3. Kill the container (docker kill) while the COB partition step is STARTED.
> 4. Start the container again and read the application log and the
> batch_job_execution and batch_step_execution tables.
>
> ❌ Actual Result
> • The job execution stays STARTED/UNKNOWN and the partition step stays
> STARTED forever.
> • StuckJobListener logs "Sleeping for a second to wait for the partitions to
> complete for job <id>" in an endless loop and never restarts the job.
> Readiness is not blocked.
> • Boot resets job.currently_running, but the execution is never closed.
> • Marking the partition FAILED does not help: the loop exits only on
> COMPLETED.
>
> ✅ Expected Result
> • A partition step that has not been updated for longer than a timeout
> (proposed: fineract.job.stuck-partition-timeout, default 10m) is marked
> FAILED, so the job is restarted and re-runs only the unfinished partitions.
> COMPLETED partitions are not re-run.
> • No endless loop, and one stuck tenant does not block stuck-job recovery for
> the tenants after it.
>
> ℹ️ Notes
> • Observed on Fineract 1.15.0, Docker, Postgres 16, single-instance topology.
> Not tested: multi-instance topology.
> • Workaround: set the STARTED step and job executions to ABANDONED and
> reboot. ABANDONED only closes the execution; it cannot be restarted.
> • is_suspended is deliberately not honoured: restarting a stuck job is crash
> recovery, not a scheduled run.
>
> 🔧 Technical Notes
> • StuckJobListener.java:17-45.
> StuckJobExecutorServiceImpl#waitUntilAllPartitionsFinished loops on
> JobExecutionRepository#getNotCompletedPartitionsCount, which counts every
> non-partitioner step with STATUS <> COMPLETED, so FAILED, STOPPED and
> ABANDONED worker steps also loop forever. The listener runs on an event
> thread and handles tenants one after another.
> • Fix: count only STARTING, STARTED and STOPPING steps; in each loop
> iteration run an UPDATE setting such steps to FAILED (EXIT_CODE FAILED,
> END_TIME now) where LAST_UPDATED is older than now minus the timeout. The
> existing updateJobStatusToFailed and restart then re-run only FAILED
> partitions (SimpleStepExecutionSplitter#shouldStart restarts only FAILED and
> STOPPED, which is why ABANDONED is not the target state).
> • A live worker updates LAST_UPDATED on every chunk commit, so it is not
> marked dead. The timeout must exceed the slowest chunk. LAST_UPDATED is
> written in the JVM time zone, so pods with mixed time zones would break the
> cutoff.
> • Test: StuckJobExecutorServiceImplTest (Mockito): count 1 then 0 calls the
> stale update with the right cutoff, then updateJobStatusToFailed, then
> restart; count 0 goes straight to restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
