Piotr Wargulak created FINERACT-3000:
----------------------------------------
Summary: Working Capital loan COB loses loans when two partitions
run at once (worker reader is not step-scoped)
Key: FINERACT-3000
URL: https://issues.apache.org/jira/browse/FINERACT-3000
Project: Apache Fineract
Issue Type: Bug
Reporter: Piotr Wargulak
{{WorkingCapitalLoanCOBWorkerConfiguration#workingCapitalLoanCOBWorkerStep}}
creates one {{WorkingCapitalLoanCOBWorkerItemReader}} and one
{{WorkingCapitalLoanCOBWorkerItemProcessor}} with {{new}} when the step bean is
built, so every partition of the job shares them. The reader's
{{remainingData}} queue is replaced in {{{}beforeStep{}}}.
Loan COB uses {{@StepScope}} beans for the same roles
({{{}LoanCOBWorkerConfiguration#cobWorkerItemReader{}}}, since 2022).
Partitions of one job can run at the same time on one instance: the
{{inboundRequests}} channel is consumed by both worker steps, and the
{{scheduling}} pool has 4 threads. When two partitions of
{{WORKING_CAPITAL_LOAN_COB_JOB}} overlap, the second one's {{beforeStep}}
replaces the first one's queue. The first partition's loans are locked
({{{}LOAN_COB_CHUNK_PROCESSING{}}}) but never read. The job ends {{COMPLETED}}
and the locks stay: the writer releases only the loans it wrote, and the unlock
step needs {{{}last_closed_business_date =
lock_placed_on_cob_business_date{}}}. From then on the scheduled run no longer
picks those loans up, and operations on them are refused.
*Steps to reproduce* (one instance, default
{{{}WC_LOAN_COB_PARTITION_SIZE=100{}}})
# Create and disburse 100 Working Capital loans, and 2 more (102 loans, so 2
partitions).
# Move the business date one day at a time and run the Working Capital Loan
COB job each day.
# Within a few nights one run logs
{{workingCapitalLoanCOBWorkerStep:partition_0}} and {{partition_1}} on two
{{scheduling-N}} threads at once; {{partition_0}} logs {{Apply locks for [1 …
100]}} and then processes loan 102.
# {{m_wc_loan_account_locks}} then holds 100 rows with no error, and those
loans' {{last_closed_business_date}} never moves again.
In our runs (32 nights each) two partitions lost loans in 3 of 3 runs, once for
good (100 loans). With one partition ({{{}WC_LOAN_COB_PARTITION_SIZE{}}} above
the number of loans) none were lost.
*Proposed fix:* make the WC worker reader, processor and writer {{@StepScope}}
beans, as Loan COB does, and have the reader implement
{{StepExecutionListener}} instead of using {{@BeforeStep}} (a step-scope proxy
hides the annotation; {{CobStepListenerRegistrationTest}} pins the interface
form).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
