[
https://issues.apache.org/jira/browse/FINERACT-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Saghy reassigned FINERACT-2995:
------------------------------------
Assignee: Adam Saghy (was: Jose Alberto Hernandez)
> WC COB worker shares reader and processor across concurrent partitions
> ----------------------------------------------------------------------
>
> Key: FINERACT-2995
> URL: https://issues.apache.org/jira/browse/FINERACT-2995
> Project: Apache Fineract
> Issue Type: Bug
> Components: Loan
> Affects Versions: 1.16.0
> Reporter: Jose Alberto Hernandez
> Assignee: Adam Saghy
> Priority: Major
> Labels: bug, cob, concurrency, spring-batch, working-capital-loan
> Fix For: 1.17.0
>
>
> ```
> h2. Background
> The Working Capital (WC) close-of-business job
> (\{{WORKING_CAPITAL_LOAN_COB_JOB}}) is a remote-partitioned Spring Batch job:
> a manager step splits the eligible loans into partitions of
> \{{WC_LOAN_COB_PARTITION_SIZE}} (default 100) and the worker step
> (\{{workingCapitalLoanCOBWorkerStep}}) processes each partition on a
> scheduling thread. Several partitions run concurrently on the same instance
> whenever a tenant has more eligible loans than the partition size, or when
> several tenants run WC COB at the same time.
> h2. Problem
> {\{WorkingCapitalLoanCOBWorkerConfiguration}} builds the chunk reader,
> processor and writer with \{{new}} inside the singleton worker step bean, so
> one instance of each serves every partition of every tenant in the JVM. These
> components hold per-partition state in instance fields:
> * \{{WorkingCapitalLoanCOBWorkerItemReader#remainingData}} - the partition's
> loan ids, filled by \{{@BeforeStep}};
> * \{{AbstractItemProcessor#executionContext}} and \{{#businessDate}} - the
> step's execution context and COB business date.
> When two partitions start at the same time their \{{@BeforeStep}} calls
> overwrite each other's queue. One partition then polls an empty queue, reads
> nothing, finishes \{{COMPLETED}} with \{{read_count = 0}}, and the job ends
> \{{COMPLETED}} with no error. The loan of the lost partition keeps its old
> \{{last_closed_business_date}}, so the regular partition query
> (\{{last_closed_business_date = COB_DATE - 1}}) never selects it again, and
> the \{{LOAN_COB_CHUNK_PROCESSING}} lock placed before the read stays in
> \{{m_wc_loan_account_locks}} because the unlock step only releases processed
> loans. Unlike LOAN_COB, WC COB has no stayed-locked step, so nothing reports
> the loan. With different business dates per tenant the shared processor could
> also stamp a loan with another tenant's COB date.
> {\{LoanCOBWorkerConfiguration}} declares the same three components as
> \{{@StepScope}} beans and does not show the problem under the same load.
> h2. Steps to reproduce
> # Single tenant on develop (observed @ 537e19aaa9), start Fineract in batch
> manager mode with \{{WC_LOAN_COB_PARTITION_SIZE=1}} so every loan is its own
> partition.
> # Create a WC product and disburse 4 WC loans on business date 01 January
> 2024.
> # For 5 days: move the business date one day forward and run the WC COB job
> (\{{POST /v1/jobs/short-name/WC_COB?command=executeJob}}).
> # Inspect \{{m_wc_loan.last_closed_business_date}},
> \{{m_wc_loan_account_locks}} and \{{BATCH_STEP_EXECUTION}} for the worker
> step.
> h2. Expected behaviour
> All 4 loans end with \{{last_closed_business_date = 2024-01-05}}, every
> worker partition has \{{read_count = 1}}, \{{m_wc_loan_account_locks}} is
> empty after each job.
> h2. Actual behaviour
> * Loans 1 and 3 end at 2024-01-05; loan 4 stays at 2024-01-02 and loan 2 at
> 2024-01-03.
> * Worker step executions 4, 19 and 27 finished \{{COMPLETED}} with
> \{{read_count = 0}}; in every case the sibling partition started on another
> \{{scheduling-N}} thread in the same millisecond.
> * \{{m_wc_loan_account_locks}} keeps rows for loans 2 and 4
> (\{{LOAN_COB_CHUNK_PROCESSING}}, COB dates 2024-01-04 and 2024-01-03).
> * 0 exceptions, 0 ERROR log lines, all 5 job executions \{{COMPLETED}}.
> The same pattern appeared earlier with 3 tenants and 2 WC loans each: 2 of 6
> loans left behind, both with a stale lock.
> h2. Proposed fix
> * Declare the WC worker reader, processor and writer as \{{@Bean @StepScope}}
> factory methods returning the concrete classes (so Spring Batch finds the
> \{{@BeforeStep}} listener on the scoped proxy's target class) and inject them
> into the worker step, mirroring \{{LoanCOBWorkerConfiguration}}.
> * Add a unit test that pins the three components to step scope
> (\{{WorkingCapitalLoanCOBWorkerStepScopeTest}}).
> * Follow-up (separate ticket): a stayed-locked step for WC COB, as LOAN_COB
> has, so a loan left locked after the partitions is reported instead of
> staying behind silently. It needs a new WC business event.
> h2. Affected modules
> * \{{fineract-working-capital-loan}} -
> \{{WorkingCapitalLoanCOBWorkerConfiguration}} (step-scoped components), new
> unit test.
> h2. Acceptance criteria
> * With \{{WC_LOAN_COB_PARTITION_SIZE=1}} and 4 eligible WC loans in one
> tenant, 5 consecutive WC COB runs leave all loans at the expected
> \{{last_closed_business_date}} and no rows in \{{m_wc_loan_account_locks}}.
> * Every worker partition step execution reports \{{read_count}} equal to the
> number of loans in its partition.
> * \{{CobSequentialProcessingTest}} still passes (no task executor introduced
> on the worker step).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
