This question is fundamentally about *DevOps* and *stack architecture* rather than core Apache Fineract application development. It addresses a general problem in distributed systems, *not a Fineract-specific bug or feature. *
However, as scheduling is a key element for the community, I suggest the proper question is: Is there a recommended Best Practice for your use case? Below are 3 common practices, there are many more. To prevent a scheduled job from running multiple times when your GKE application scales across several pods, you should employ a distributed locking mechanism or a centralized scheduling approach. Here are the recommended strategies: 1. Implement Distributed Locking A distributed lock ensures that only one instance of your application can acquire the lock and execute the scheduled job at any given time. There are multiple ways to implement locks, but *Database Locking is frequently used*: You can use a shared database to manage locks. Before a job runs, each pod attempts to acquire a lock by creating a specific entry in a lock table with a unique constraint. The pod that successfully creates the entry proceeds with the job. 2. Leverage Quartz Scheduler Clustering Feature If using Quartz, know that it may be configure it to work in a clustered mode. This feature is specifically designed for high availability and to prevent duplicate job executions in a multi-node environment. *JDBC JobStore: To enable clustering, you must configure Quartz to use the JDBC-JobStore. All pods (nodes in the cluster) will point to the same database. Quartz uses database-level locking (SELECT ... FOR UPDATE) to ensure that only one node fires a given trigger. For more information, refer to the Quartz Scheduler documentation on clustering <https://www.quartz-scheduler.org/documentation/quartz-2.3.0/configuration/ConfigJDBCJobStoreClustering.html>. *3. Implement Leader Election In this practice, one pod is elected as the "leader." Only the leader pod is responsible for executing the scheduled jobs. The other pods remain on standby and can take over if the leader fails. *Kubernetes Leader Election*: You can implement leader election using Kubernetes APIs. The client-go library for Go applications and other language-specific libraries provide abstractions for this. The leader's identity is typically stored in a Lease or ConfigMap object within the Kubernetes cluster. See the Kubernetes documentation on Simple leader election with Kubernetes <https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/> . Please ensure questions posted here clearly distinguish between *Fineract issues* and *stack-related problems*. Being precise helps the community provide effective support or direct to where support would be more appropriate. On Fri, Jul 4, 2025 at 4:04 AM Bhaskar Tiwari <bhas...@strideone.in> wrote: > Hi, > > > > I’d appreciate your guidance on an additional scenario related to our GKE > deployment. Specifically, when auto-scaling in GCP creates multiple pods > running the same application, each pod ends up with its own instance of the > Quartz Scheduler. This results in the same scheduled job being executed > multiple times in parallel — once by each pod — since they all share the > same configuration. > > Could you please advise on best practices or strategies to ensure that the > scheduled job runs only once, even when the application is scaled across > multiple pods? > > > > *From:* Bhaskar Tiwari > *Sent:* Thursday, July 3, 2025 7:37 PM > *To:* dev@fineract.apache.org > *Subject:* RE: Handling Duplicate Entries from Custom Schedulers in > Fineract (Multi-Server Environment) > > > > Thanks a lot for the response. > > > > Can you please guide on below scenario also: > > > > What happens when auto-scaling creates multiple pods in GCP (Kubernetes) > that are running the same application. > > GKE (Google Kubernetes Engine) automatically scales the number of pods > based on traffic/resource usage. > > > > If multiple pods are created via auto-scaling and each pod has its own > Quartz Scheduler, then each pod will try to execute the job independently > causing the same job to run multiple times because they share the same > config file. > > > > How to Avoid Running the Same Job Multiple Times for above case? > > > > *From:* Ádám Sághy <adamsa...@gmail.com> > *Sent:* Thursday, July 3, 2025 1:56 PM > *To:* dev@fineract.apache.org > *Subject:* Re: Handling Duplicate Entries from Custom Schedulers in > Fineract (Multi-Server Environment) > > > > Hi > > > > Victor was faster than me, but yes, he is right. > > > > You can set the “fineract.node-id” in the application.properties directly > (not recommended) or override as environement variable (FINERACT_NODE_ID) ( > *recommended*). > > > > In the “job” table you can find a column for each job that exists in the > system: “node_id” this value is matched with the scheduler on the running > instances and if the running instances node_id is matching that particular > scheduler will execute the job. > > > > Make sure you are not running multiple instance of Fineract with the same > “node_id” or that lead to duplicate job execution! > > > > I hope this helps! > > > > Regards, > > Adam > > > > On 2025. Jul 3., at 7:53, VICTOR MANUEL ROMERO RODRIGUEZ < > victor.rom...@fintecheando.mx> wrote: > > > > Bashar, > > > > The variable can be set in your environment, some samples: > > > > > https://github.com/search?q=repo%3Aapache%2Ffineract%20FINERACT_NODE_ID&type=code > > > > El mié, 2 jul 2025 a las 23:46, Bhaskar Tiwari (<bhas...@strideone.in>) > escribió: > > Hi, > > > > Any update on this? > > > > *From:* Bhaskar Tiwari > *Sent:* Monday, June 30, 2025 4:32 PM > *To:* dev@fineract.apache.org > *Subject:* RE: Handling Duplicate Entries from Custom Schedulers in > Fineract (Multi-Server Environment) > > > > Hi Adam, > > > > Sorry for the confusion earlier — to clarify, we are indeed using the > Quartz scheduler as provided by Fineract, with some custom logic added for > new modules. So, we're not using a separate scheduler framework; it's just > an extension of the existing Quartz-based implementation. > > Regarding your suggestion on using node_id: 1, could you please guide us > on where exactly we need to configure this? Specifically: > > > > Should the node_id be set as a Java system property or as part of a Spring > configuration? > > > > Is there any additional setup needed in the “job” table or related > components to ensure only the intended instance runs the schedulers? > > > > We want to make sure we implement this correctly to avoid any further > issues in a multi-server environment. > > > > Thanks again for your guidance. > > > > *From:* Ádám Sághy <adamsa...@gmail.com> > *Sent:* Monday, June 30, 2025 3:27 PM > *To:* dev@fineract.apache.org > *Subject:* Re: Handling Duplicate Entries from Custom Schedulers in > Fineract (Multi-Server Environment) > > > > Hi > > > > Thank you for the additional details. > > > > 1. I am not completely sure why you need “custom” scheduler for this > purpose… quartz should be fine to be used…unless you know something that > was not yet disclosed... > > 2. You should not run schedulers on 2 instances… only 1 instance should do > the scheduling, so there wont be conflicts! In Fineract the quartz > scheduler is running on “node_id: 1” and only 1 instance should have this > configuration! > > > > I hope this helps! > > > > Regards, > > Adam > > > > On 2025. Jun 30., at 11:45, Bhaskar Tiwari <bhas...@strideone.in> wrote: > > > > Hi Ádám Sághy > > > > Thank you for your response. > > > > Please find below the clarification on your queries: > > > > 1. *What exactly do you mean by “custom scheduler”?* > > We have developed a few custom schedulers to enhance Fineract’s > functionality, specifically for new modules that require generating daily > accruals. These schedulers follow the existing scheduler patterns within > Fineract but are tailored for our custom logic. > > > > 2. *How are the schedulers deployed across multiple servers?* > > We have two servers running the same codebase, both connected to the same > database. Each server has the scheduler enabled. The job configurations, > including the cron_expression, are stored in the shared job table in the > database. As a result, both servers read the job configuration and > independently trigger the same scheduled tasks, leading to duplicate > execution. > > > > 3. *Are they part of the same application instance or running > independently?* > > The schedulers are running on two separate, independent application > instances hosted on different servers. Each instance reads from the same > database and executes the scheduler jobs individually, which is causing the > duplication issue. > > > > Please let me know if you need any further details or logs. I’d appreciate > any suggestions on how best to manage schedulers in such a multi-server > environment to prevent duplicate job executions. > > > > *From:* Ádám Sághy <adamsa...@gmail.com> > *Sent:* Monday, June 30, 2025 2:32 PM > *To:* dev@fineract.apache.org > *Subject:* Re: Handling Duplicate Entries from Custom Schedulers in > Fineract (Multi-Server Environment) > > > > Hi Bhaskar, > > Thank you for reaching out. > > I’d appreciate it if you could clarify a few points to help me better > understand the issue: > > - What exactly do you mean by “custom scheduler”? > - How are the schedulers deployed across multiple servers? > - Are they part of the same application instance or running > independently? > > A bit more detail on your setup and the logic you’ve implemented would be > helpful in identifying the root cause and suggesting a solution. > > Regards, > Adam > > > > On 2025. Jun 30., at 10:11, Bhaskar Tiwari <bhas...@strideone.in> wrote: > > > > Hi Team, > > > > I’ve implemented some custom schedulers in Fineract using existing > patterns. However, I’m encountering an issue with duplicate entries when > two servers are running concurrently and share the same database. > > While I’ve added logic to prevent duplicate accrual entries when > schedulers are triggered multiple times, the problem arises because these > schedulers are running on separate servers. As a result, each instance > bypasses the duplication check, leading to duplicate accrual entries in the > database. > > Is there a recommended way to manage custom schedulers in a multi-server > setup to avoid this issue? > > > > Looking forward to your suggestions. > > > > Regards > > *Bhaskar Tiwari* > > > > *"Print this mail only if absolutely necessary. Save Paper. Save Trees."* > *Disclaimer: *“This electronic mail message sent from StrideOne (Stride > Fintree Private Limited) may contain Confidential/Restricted/Internal > information and should only be viewed by the intended recipients. Under no > circumstances may any such information be disclosed, copied, used or > distributed to any unauthorized persons or entities without the written > consent of Strideone. If you are not the intended recipient, any review, > retransmission, dissemination or reliance on the content of these materials > is strictly prohibited and may be the subject of legal action. If you > received this email in error, please notify the sender and delete the > message immediately.” > > > > > > *"Print this mail only if absolutely necessary. Save Paper. Save Trees."* > *Disclaimer: *“This electronic mail message sent from StrideOne (Stride > Fintree Private Limited) may contain Confidential/Restricted/Internal > information and should only be viewed by the intended recipients. Under no > circumstances may any such information be disclosed, copied, used or > distributed to any unauthorized persons or entities without the written > consent of Strideone. If you are not the intended recipient, any review, > retransmission, dissemination or reliance on the content of these materials > is strictly prohibited and may be the subject of legal action. If you > received this email in error, please notify the sender and delete the > message immediately.” > > > > > > *"Print this mail only if absolutely necessary. Save Paper. Save Trees."* > *Disclaimer: > *“This electronic mail message sent from StrideOne (Stride Fintree > Private Limited) may contain Confidential/Restricted/Internal information > and should only be viewed by the intended recipients. Under no > circumstances may any such information be disclosed, copied, used or > distributed to any unauthorized persons or entities without the written > consent of Strideone. If you are not the intended recipient, any review, > retransmission, dissemination or reliance on the content of these materials > is strictly prohibited and may be the subject of legal action. If you > received this email in error, please notify the sender and delete the > message immediately.” > > > > > *"Print this mail only if absolutely necessary. Save Paper. Save Trees."* * > Disclaimer: *“This electronic mail message sent from StrideOne (Stride > Fintree Private Limited) may contain Confidential/Restricted/Internal > information and should only be viewed by the intended recipients. Under no > circumstances may any such information be disclosed, copied, used or > distributed to any unauthorized persons or entities without the written > consent of Strideone. If you are not the intended recipient, any review, > retransmission, dissemination or reliance on the content of these materials > is strictly prohibited and may be the subject of legal action. If you > received this email in error, please notify the sender and delete the > message immediately.” > -- -- Paul
