[
http://mifosforge.jira.com/browse/MIFOS-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58362#action_58362
]
johnwoodlock commented on MIFOS-2986:
-------------------------------------
Info for Jeff:
Noticed slow performance on the following size dataset (local machine)
Customers: 220K
Customer Schedules: 18.5M
Loan Schedules: 4.2M
Savings Schedules: 993
I had done work on this batch job previously and knew that it had been updated
for office holidays (and moratoriums) so did a check. The processing involved
does depend on the no. of holidays added and the dates of the holidays and the
type of holiday (moratoriums are the biggest impact) but the following is
representative of what I saw. Adding holidays is a rare event but I think
still needs to run in a sensible timeframe.
Updates to Customer Schedules were taking over 2 minutes per 1000 clients.
(never got to loan schedules). So if you added a holiday somewhere in the next
few weeks and you had 500k clients it would run for over 1000 mins (16 hours)
and possibly much longer for loan schedules. I knew this was way more than it
was previously. As some of the issue was running the same query many times, I
ran the job with 'query-caching' enabled on mysql which reduced the customer
schedule time to 76 seconds per 1000 clients but even that (over 10 hours for
this bit) is too much.
After this fix customer schedules were about 11 seconds per 1000 clients and
loan schedules about 25 seconds per 1000 clients. This would equate to about
1.5 hrs for customer schedules and 3.5 hrs for loan schedules - 5 hrs is still
hefty but manageable whereas the previous equivalent (10 + 23 = 33 hours would
cause problems.
> ApplyHolidayChangesTask batch task needs tweaking now that it handles branch
> level holiday (performance/scalability)
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MIFOS-2986
> URL: http://mifosforge.jira.com/browse/MIFOS-2986
> Project: mifos
> Issue Type: Bug
> Components: Batch jobs
> Affects Versions: Shamim D
> Reporter: johnwoodlock
> Assignee: johnwoodlock
> Priority: Major
> Fix For: Shamim D
>
> Attachments: AccountPersistence.java, ApplyHolidayChangesHelper.java
>
>
> Changes to the code to allow the batch job to handle branch level holidays
> have slowed it down a lot.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://mifosforge.jira.com/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
Mifos-issues mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mifos-issues
