Jose Alberto Hernandez created FINERACT-2995:
------------------------------------------------
Summary: WC COB worker shares reader and processor across
concurrent partitions
Key: FINERACT-2995
URL: https://issues.apache.org/jira/browse/FINERACT-2995
Project: Apache Fineract
Issue Type: Bug
Components: Loan
Reporter: Jose Alberto Hernandez
Fix For: 1.17.0
```
h2. Background
The Working Capital (WC) close-of-business job
(\{{WORKING_CAPITAL_LOAN_COB_JOB}}) is a remote-partitioned Spring Batch job: a
manager step splits the eligible loans into partitions of
\{{WC_LOAN_COB_PARTITION_SIZE}} (default 100) and the worker step
(\{{workingCapitalLoanCOBWorkerStep}}) processes each partition on a scheduling
thread. Several partitions run concurrently on the same instance whenever a
tenant has more eligible loans than the partition size, or when several tenants
run WC COB at the same time.
h2. Problem
{\{WorkingCapitalLoanCOBWorkerConfiguration}} builds the chunk reader,
processor and writer with \{{new}} inside the singleton worker step bean, so
one instance of each serves every partition of every tenant in the JVM. These
components hold per-partition state in instance fields:
* \{{WorkingCapitalLoanCOBWorkerItemReader#remainingData}} - the partition's
loan ids, filled by \{{@BeforeStep}};
* \{{AbstractItemProcessor#executionContext}} and \{{#businessDate}} - the
step's execution context and COB business date.
When two partitions start at the same time their \{{@BeforeStep}} calls
overwrite each other's queue. One partition then polls an empty queue, reads
nothing, finishes \{{COMPLETED}} with \{{read_count = 0}}, and the job ends
\{{COMPLETED}} with no error. The loan of the lost partition keeps its old
\{{last_closed_business_date}}, so the regular partition query
(\{{last_closed_business_date = COB_DATE - 1}}) never selects it again, and the
\{{LOAN_COB_CHUNK_PROCESSING}} lock placed before the read stays in
\{{m_wc_loan_account_locks}} because the unlock step only releases processed
loans. Unlike LOAN_COB, WC COB has no stayed-locked step, so nothing reports
the loan. With different business dates per tenant the shared processor could
also stamp a loan with another tenant's COB date.
{\{LoanCOBWorkerConfiguration}} declares the same three components as
\{{@StepScope}} beans and does not show the problem under the same load.
h2. Steps to reproduce
# Single tenant on develop (observed @ 537e19aaa9), start Fineract in batch
manager mode with \{{WC_LOAN_COB_PARTITION_SIZE=1}} so every loan is its own
partition.
# Create a WC product and disburse 4 WC loans on business date 01 January 2024.
# For 5 days: move the business date one day forward and run the WC COB job
(\{{POST /v1/jobs/short-name/WC_COB?command=executeJob}}).
# Inspect \{{m_wc_loan.last_closed_business_date}},
\{{m_wc_loan_account_locks}} and \{{BATCH_STEP_EXECUTION}} for the worker step.
h2. Expected behaviour
All 4 loans end with \{{last_closed_business_date = 2024-01-05}}, every worker
partition has \{{read_count = 1}}, \{{m_wc_loan_account_locks}} is empty after
each job.
h2. Actual behaviour
* Loans 1 and 3 end at 2024-01-05; loan 4 stays at 2024-01-02 and loan 2 at
2024-01-03.
* Worker step executions 4, 19 and 27 finished \{{COMPLETED}} with
\{{read_count = 0}}; in every case the sibling partition started on another
\{{scheduling-N}} thread in the same millisecond.
* \{{m_wc_loan_account_locks}} keeps rows for loans 2 and 4
(\{{LOAN_COB_CHUNK_PROCESSING}}, COB dates 2024-01-04 and 2024-01-03).
* 0 exceptions, 0 ERROR log lines, all 5 job executions \{{COMPLETED}}.
The same pattern appeared earlier with 3 tenants and 2 WC loans each: 2 of 6
loans left behind, both with a stale lock.
h2. Proposed fix
* Declare the WC worker reader, processor and writer as \{{@Bean @StepScope}}
factory methods returning the concrete classes (so Spring Batch finds the
\{{@BeforeStep}} listener on the scoped proxy's target class) and inject them
into the worker step, mirroring \{{LoanCOBWorkerConfiguration}}.
* Add a unit test that pins the three components to step scope
(\{{WorkingCapitalLoanCOBWorkerStepScopeTest}}).
* Follow-up (separate ticket): a stayed-locked step for WC COB, as LOAN_COB
has, so a loan left locked after the partitions is reported instead of staying
behind silently. It needs a new WC business event.
h2. Affected modules
* \{{fineract-working-capital-loan}} -
\{{WorkingCapitalLoanCOBWorkerConfiguration}} (step-scoped components), new
unit test.
h2. Acceptance criteria
* With \{{WC_LOAN_COB_PARTITION_SIZE=1}} and 4 eligible WC loans in one tenant,
5 consecutive WC COB runs leave all loans at the expected
\{{last_closed_business_date}} and no rows in \{{m_wc_loan_account_locks}}.
* Every worker partition step execution reports \{{read_count}} equal to the
number of loans in its partition.
* \{{CobSequentialProcessingTest}} still passes (no task executor introduced on
the worker step).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
