[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Philip Orr changed: What|Removed |Added Text to go in the||This enhancement adds the release notes||ability to increase the ||amount of parallel ||calculations used to create ||the Holds Queue. The amount ||of calculations or "loops" ||done in parallel is set in ||the new system preference ||'HoldsQueueParallelLoopsCou ||nt'. This enhancement will ||allow faster holds queue ||building for libraries with ||a very large amount of ||holds. Note that increasing ||the value in the system ||preference also causes Koha ||to use more resources when ||building the holds queue. Keywords|release-notes-needed| CC||philip@lmscloud.de --- Comment #36 from Philip Orr --- added release notes -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #35 from Katrin Fischer --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Katrin Fischer changed: What|Removed |Added Version(s)||24.11.00 released in|| Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Katrin Fischer changed: What|Removed |Added Keywords||release-notes-needed Version|unspecified |Main --- Comment #34 from Katrin Fischer --- Wow, old style database update, reformatted to use the newer template on review. I verified that the holds queue is still built running the script, but didn't test the parallel processing. Trusting the process :) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) changed: What|Removed |Added Status|Signed Off |Passed QA --- Comment #33 from Martin Renvoize (ashimema) --- This works well and I see no regressions. Passing QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) changed: What|Removed |Added Attachment #173046|0 |1 is obsolete|| --- Comment #32 from Martin Renvoize (ashimema) --- Created attachment 173262 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173262&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t Signed-off-by: Martin Renvoize -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) changed: What|Removed |Added Attachment #173045|0 |1 is obsolete|| --- Comment #31 from Kyle M Hall (khall) --- Created attachment 173046 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173046&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) changed: What|Removed |Added Attachment #167349|0 |1 is obsolete|| --- Comment #30 from Kyle M Hall (khall) --- Created attachment 173045 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173045&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) changed: What|Removed |Added Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #29 from Martin Renvoize (ashimema) --- Is this one still relevant Kyle? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize changed: What|Removed |Added CC|martin.renvoize@ptfs-europe | |.com| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Nind changed: What|Removed |Added Status|Needs Signoff |Patch doesn't apply --- Comment #28 from David Nind --- The patch no longer applies for me 8-(... git bz apply 28833 Bug 28833 - Speed up holds queue builder via parallel processing 167349 - Bug 28833: Speed up holds queue builder via parallel processing Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 28833: Speed up holds queue builder via parallel processing Using index info to reconstruct a base tree... M C4/HoldsQueue.pm M debian/templates/koha-conf-site.xml.in M etc/koha-conf.xml M misc/cronjobs/holds/build_holds_queue.pl Falling back to patching base and 3-way merge... Auto-merging misc/cronjobs/holds/build_holds_queue.pl CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl Auto-merging etc/koha-conf.xml Auto-merging debian/templates/koha-conf-site.xml.in Auto-merging C4/HoldsQueue.pm CONFLICT (content): Merge conflict in C4/HoldsQueue.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel processing -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Brendan Lawlor changed: What|Removed |Added CC||blaw...@clamsnet.org --- Comment #27 from Brendan Lawlor --- Test notes: > This script will generate 100 holds: > > #!/usr/bin/perl > use t::lib::TestBuilder; > my $builder = t::lib::TestBuilder->new; > > for ( my $i = 0 ; $i < 100 ; $i++ ) { > my $hold = $builder->build( { source => 'Reserve' } ); > } > I ran this script 10+ times. I noticed that in addition to holds it also makes new patrons, branches, bibs and itemtypes, probably other stuff too. cool! I could see 1000+ records in the reserves table, but if I went to a patron record where a hold was created by the script the holds table wouldn't load, with error 500. I wasn't able to build the holds queue. Also the bib detail page returned the error: DBIC result _type isn't of the _type BiblioMetadata at /kohadevbox/koha/catalogue/detail.pl line 105. at /usr/lib/x86_64-linux-gnu/perl-base/Carp.pm line 289 I think the next step needs to be to reindex for the new bibs, so I tried: koha-rebuild-zebra -f --force -v kohadev Something is not working to reindex the bibs created by the script though, once it gets past the ktd default bibs it returns errors like .error retrieving biblio 3000 at /kohadevbox/koha/misc/migration_tools/rebuild_zebra.pl line 706. Am I missing a step in the reindexing part? What step/s are required after generating the test data? I was able to install Parallel::ForkManager and use the time utility, so I will be able to test again once I can resolve the data issue. Thanks! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize changed: What|Removed |Added QA Contact|testo...@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com CC||martin.renvoize@ptfs-europe ||.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #26 from Kyle M Hall --- (In reply to David Nind from comment #25) > Thanks Kyle for making the patch apply! > > Is there an easy way to generate a large number of holds? Either an SQL > query or a script. > > David This script will generate 100 holds: #!/usr/bin/perl use t::lib::TestBuilder; my $builder = t::lib::TestBuilder->new; for ( my $i = 0 ; $i < 100 ; $i++ ) { my $hold = $builder->build( { source => 'Reserve' } ); } Hope this helps! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #25 from David Nind --- Thanks Kyle for making the patch apply! Is there an easy way to generate a large number of holds? Either an SQL query or a script. David -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #126216|0 |1 is obsolete|| --- Comment #24 from Kyle M Hall --- Created attachment 167349 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167349&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Nind changed: What|Removed |Added Status|Needs Signoff |Patch doesn't apply CC||da...@davidnind.com --- Comment #23 from David Nind --- As expected, since this has been in the sign off queue for a while, the patch no longer applies 8-(... Is there a way to generate a large number of holds? Either an SQL query or a script. kohadev-koha@kohadevbox:koha(bz28833)$ git bz apply 28833 Bug 28833 - Speed up holds queue builder via parallel processing 126216 - Bug 28833: Speed up holds queue builder via parallel processing Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 28833: Speed up holds queue builder via parallel processing Using index info to reconstruct a base tree... M C4/HoldsQueue.pm M debian/templates/koha-conf-site.xml.in M etc/koha-conf.xml M misc/cronjobs/holds/build_holds_queue.pl Falling back to patching base and 3-way merge... Auto-merging misc/cronjobs/holds/build_holds_queue.pl CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl Auto-merging etc/koha-conf.xml CONFLICT (content): Merge conflict in etc/koha-conf.xml Auto-merging debian/templates/koha-conf-site.xml.in CONFLICT (content): Merge conflict in debian/templates/koha-conf-site.xml.in Auto-merging C4/HoldsQueue.pm CONFLICT (content): Merge conflict in C4/HoldsQueue.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel processing -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #126202|0 |1 is obsolete|| --- Comment #22 from Kyle M Hall --- Created attachment 126216 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126216&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123996|0 |1 is obsolete|| Attachment #126200|0 |1 is obsolete|| --- Comment #21 from Kyle M Hall --- Created attachment 126202 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126202&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #20 from Kyle M Hall --- Created attachment 126200 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126200&action=edit Bug 28833: Balance the number of holds each chunk contains -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #19 from Kyle M Hall --- (In reply to David Cook from comment #17) > Comment on attachment 123689 [details] [review] > Bug 28833: Speed up holds queue builder via parallel processing > > Review of attachment 123689 [details] [review]: > - > > ::: C4/HoldsQueue.pm > @@ +210,5 @@ > > + > > +DATA_LOOP: > > +foreach my $chunk (@chunks) { > > +my $pid = $pm->start and next DATA_LOOP; > > +_ProcessBiblios($chunk); > > I notice that the other _ProcessBiblios has arguments for $branches_to_use > and $transport_cost_matrix. I'm guessing those should be included here too? Good catch! I've fixed that issue and squashed the patches down for ease of use. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123689|0 |1 is obsolete|| Attachment #123785|0 |1 is obsolete|| Attachment #123786|0 |1 is obsolete|| --- Comment #18 from Kyle M Hall --- Created attachment 123996 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123996&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #17 from David Cook --- Comment on attachment 123689 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689 Bug 28833: Speed up holds queue builder via parallel processing Review of attachment 123689: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123689) - ::: C4/HoldsQueue.pm @@ +210,5 @@ > + > +DATA_LOOP: > +foreach my $chunk (@chunks) { > +my $pid = $pm->start and next DATA_LOOP; > +_ProcessBiblios($chunk); I notice that the other _ProcessBiblios has arguments for $branches_to_use and $transport_cost_matrix. I'm guessing those should be included here too? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #16 from David Cook --- Comment on attachment 123786 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786 Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified Review of attachment 123786: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123786) - ::: C4/HoldsQueue.pm @@ +201,4 @@ > # Split the list of bibs into groups to run in parallel > if ( $loops > 1 ) { > my $bibs_per_chunk = ceil( scalar @$bibs_with_pending_requests / > $loops ); > +$bibs_per_chunk = @$bibs_with_pending_requests if > @$bibs_with_pending_requests <= $bibs_per_chunk; I don't understand this logic. scalar @$bibs_with_pending_requests = 10 $loops = 10 $bibs_per_chunk = 1 $bibs_per_chunk would still be 1. Maybe I'm confused but I can't think of a case where $bibs_per_chunk could be greater than scalar @$bibs_with_pending_requests. -- I was thinking we'd do the $loops would be changed to $max_loops, and $loops would be based on an arbitrary ratio. Perhaps 1000 bibs per 1 loop. So $loops = scalar @$bibs_with_pending_requests / 1000. If $loops < 1, then we don't fork. If $loops > 1, we double-check that we're not exceeding our maximum number of loops. For example: $max_loops = 4 scalar @$bibs_with_pending_requests = 1 $loops = 10 We throttle $loops down from 10 to 4, because 4 is our maximum. Likewise, if @$bibs_with_pending_requests = 2000, we're only using 2 forks instead of our maximum of 4, so we're not using more resources than necessary. -- Alternatively, we could have an arbitrary threshold of 1000. Below that and we use $loops = 1. Above that, we use the $loops value supplied from config. I'm just trying to avoid scenarios where you have 50 Koha instances each spinning up 10 child processes to process 2 requests each. But maybe I'm just prematurely optimizing. I suppose most instances will have the default of 1 anyway. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #15 from Kyle M Hall --- Created attachment 123786 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786&action=edit Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123783|0 |1 is obsolete|| Attachment #123784|0 |1 is obsolete|| --- Comment #14 from Kyle M Hall --- Created attachment 123785 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123785&action=edit Bug 28833: (follow-up) Move loops count control to koha conf -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #13 from Kyle M Hall --- Created attachment 123784 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123784&action=edit Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #12 from Kyle M Hall --- Created attachment 123783 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123783&action=edit Bug 28833: (follow-up) Move loops count control to koha conf -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #11 from David Cook --- The following is totally out of scope but inspired by the above: What if default values in koha-conf.xml were determined based on a "profile" provided for koha-create? By default, we'd have permissive values, so that inexperienced people could have the most power possible through the Web UI, since they might not have a sysadmin to update the backend. But we'd have a more restrictive profile that vendors and more experienced people could use to have more locked down/secure Koha instances. I'm trying to think of an approach that could make everyone happy... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #10 from David Cook --- Another option might be to use koha-conf.xml instead of a system preference. I think it's important for it to be a configurable value per-instance, but I'm opposed to the idea of library staff being able to determine server resource usage. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Cook changed: What|Removed |Added CC||dc...@prosentient.com.au --- Comment #9 from David Cook --- This is an interesting one! On one hand, I like the use of a system preference to allow the value to be set on a per-instance basis. On the other hand, letting a library define an unrestricted number of forks for a script seems potentially dangerous, especially if you have multiple Koha instances on the same server. I think this change needs more sanity checks. Some options: 1) A CLI override that doesn't let the $loops be beyond X. This allows a sysadmin to constrain the resource usage 2) Perhaps the size of $bibs_with_pending_requests should correlate with the number of forks up to a maximum. (If you only have 10 bibs but your $loops is 10, it doesn't make sense to have 10 forks of 1 bib each. ) 3) Maybe we should limit nproc in /etc/security/limits.d on a per-user basis (but maybe this is best left to local sysadmins and this is a very blunt instrument) Actually, we should also do a int() check on $loops before feeding it to Parallel::ForkManager, since it is user-provided input. (I suppose a non-number would just cause a fatal error on a $loops > 1 check, but better to not fork than to fail.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123688|0 |1 is obsolete|| --- Comment #8 from Kyle M Hall --- Created attachment 123689 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123679|0 |1 is obsolete|| --- Comment #7 from Kyle M Hall --- Created attachment 123688 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123688&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123671|0 |1 is obsolete|| --- Comment #6 from Kyle M Hall --- Created attachment 123679 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123679&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::Loops 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123659|0 |1 is obsolete|| --- Comment #5 from Kyle M Hall --- Created attachment 123671 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123671&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Jessie Zairo changed: What|Removed |Added CC||jza...@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123657|0 |1 is obsolete|| --- Comment #4 from Kyle M Hall --- Created attachment 123659 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123659&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Attachment #123656|0 |1 is obsolete|| --- Comment #3 from Kyle M Hall --- Created attachment 123657 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123657&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #2 from Kyle M Hall --- This patch provides a tremendous speedup for large numbers of holds. However, it causes HoldsQueue.t to fail spectacularly. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #1 from Kyle M Hall --- Created attachment 123656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123656&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall changed: What|Removed |Added Assignee|koha-b...@lists.koha-commun |k...@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/