[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-11-08 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Philip Orr  changed:

   What|Removed |Added

  Text to go in the||This enhancement adds the
  release notes||ability to increase the
   ||amount of parallel
   ||calculations used to create
   ||the Holds Queue. The amount
   ||of calculations or "loops"
   ||done in parallel is set in
   ||the new system preference
   ||'HoldsQueueParallelLoopsCou
   ||nt'. This enhancement will
   ||allow faster holds queue
   ||building for libraries with
   ||a very large amount of
   ||holds. Note that increasing
   ||the value in the system
   ||preference also causes Koha
   ||to use more resources when
   ||building the holds queue.
   Keywords|release-notes-needed|
 CC||philip@lmscloud.de

--- Comment #36 from Philip Orr  ---
added release notes

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #35 from Katrin Fischer  ---
Pushed for 24.11!

Well done everyone, thank you!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Katrin Fischer  changed:

   What|Removed |Added

 Version(s)||24.11.00
released in||
 Status|Passed QA   |Pushed to main

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Katrin Fischer  changed:

   What|Removed |Added

   Keywords||release-notes-needed
Version|unspecified |Main

--- Comment #34 from Katrin Fischer  ---
Wow, old style database update, reformatted to use the newer template on
review.
I verified that the holds queue is still built running the script, but didn't
test the parallel processing. Trusting the process :)

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Martin Renvoize (ashimema)  changed:

   What|Removed |Added

 Status|Needs Signoff   |Signed Off

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Martin Renvoize (ashimema)  changed:

   What|Removed |Added

 Status|Signed Off  |Passed QA

--- Comment #33 from Martin Renvoize (ashimema) 
 ---
This works well and I see no regressions.

Passing QA

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-24 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Martin Renvoize (ashimema)  changed:

   What|Removed |Added

 Attachment #173046|0   |1
is obsolete||

--- Comment #32 from Martin Renvoize (ashimema) 
 ---
Created attachment 173262
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173262&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

Signed-off-by: Martin Renvoize 

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-21 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall (khall)  changed:

   What|Removed |Added

 Attachment #173045|0   |1
is obsolete||

--- Comment #31 from Kyle M Hall (khall)  ---
Created attachment 173046
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173046&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-21 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall (khall)  changed:

   What|Removed |Added

 Attachment #167349|0   |1
is obsolete||

--- Comment #30 from Kyle M Hall (khall)  ---
Created attachment 173045
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173045&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-21 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall (khall)  changed:

   What|Removed |Added

 Status|Patch doesn't apply |Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-10-18 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #29 from Martin Renvoize (ashimema) 
 ---
Is this one still relevant Kyle?

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-07-25 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Martin Renvoize  changed:

   What|Removed |Added

 CC|martin.renvoize@ptfs-europe |
   |.com|

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-07-04 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

David Nind  changed:

   What|Removed |Added

 Status|Needs Signoff   |Patch doesn't apply

--- Comment #28 from David Nind  ---
The patch no longer applies for me 8-(...

git bz apply 28833

Bug 28833 - Speed up holds queue builder via parallel processing

167349 - Bug 28833: Speed up holds queue builder via parallel processing

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 28833: Speed up holds queue builder via parallel processing
Using index info to reconstruct a base tree...
M   C4/HoldsQueue.pm
M   debian/templates/koha-conf-site.xml.in
M   etc/koha-conf.xml
M   misc/cronjobs/holds/build_holds_queue.pl
Falling back to patching base and 3-way merge...
Auto-merging misc/cronjobs/holds/build_holds_queue.pl
CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl
Auto-merging etc/koha-conf.xml
Auto-merging debian/templates/koha-conf-site.xml.in
Auto-merging C4/HoldsQueue.pm
CONFLICT (content): Merge conflict in C4/HoldsQueue.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel
processing

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-07-02 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Brendan Lawlor  changed:

   What|Removed |Added

 CC||blaw...@clamsnet.org

--- Comment #27 from Brendan Lawlor  ---
Test notes:

 > This script will generate 100 holds:
> 
> #!/usr/bin/perl
> use t::lib::TestBuilder;
> my $builder = t::lib::TestBuilder->new;
> 
> for ( my $i = 0 ; $i < 100 ; $i++ ) {
> my $hold = $builder->build( { source => 'Reserve' } );
> }
> 

I ran this script 10+ times. I noticed that in addition to holds it also makes
new patrons, branches, bibs and itemtypes, probably other stuff too. cool!

I could see 1000+ records in the reserves table, but if I went to a patron
record where a hold was created by the script the holds table wouldn't load,
with error 500. I wasn't able to build the holds queue.

Also the bib detail page returned the error:
DBIC result _type  isn't of the _type BiblioMetadata at
/kohadevbox/koha/catalogue/detail.pl line 105.
 at /usr/lib/x86_64-linux-gnu/perl-base/Carp.pm line 289

I think the next step needs to be to reindex for the new bibs, so I tried:
koha-rebuild-zebra -f --force -v kohadev

Something is not working to reindex the bibs created by the script though, once
it gets past the ktd default bibs it returns errors like
.error retrieving biblio 3000 at
/kohadevbox/koha/misc/migration_tools/rebuild_zebra.pl line 706.

Am I missing a step in the reindexing part? What step/s are required after
generating the test data?

I was able to install Parallel::ForkManager and use the time utility, so I will
be able to test again once I can resolve the data issue. Thanks!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-07-02 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Martin Renvoize  changed:

   What|Removed |Added

 QA Contact|testo...@bugs.koha-communit |martin.renvoize@ptfs-europe
   |y.org   |.com
 CC||martin.renvoize@ptfs-europe
   ||.com

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-06-04 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #26 from Kyle M Hall  ---
(In reply to David Nind from comment #25)
> Thanks Kyle for making the patch apply!
> 
> Is there an easy way to generate a large number of holds? Either an SQL
> query or a script.
> 
> David

This script will generate 100 holds:

#!/usr/bin/perl
use t::lib::TestBuilder;
my $builder = t::lib::TestBuilder->new;

for ( my $i = 0 ; $i < 100 ; $i++ ) {
my $hold = $builder->build( { source => 'Reserve' } );
}

Hope this helps!

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-06-03 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #25 from David Nind  ---
Thanks Kyle for making the patch apply!

Is there an easy way to generate a large number of holds? Either an SQL query
or a script.

David

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-06-03 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #126216|0   |1
is obsolete||

--- Comment #24 from Kyle M Hall  ---
Created attachment 167349
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167349&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2024-06-03 Thread bugzilla-daemon--- via Koha-bugs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Status|Patch doesn't apply |Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2023-10-22 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

David Nind  changed:

   What|Removed |Added

 Status|Needs Signoff   |Patch doesn't apply
 CC||da...@davidnind.com

--- Comment #23 from David Nind  ---
As expected, since this has been in the sign off queue for a while, the patch
no longer applies 8-(...

Is there a way to generate a large number of holds? Either an SQL query or a
script.


kohadev-koha@kohadevbox:koha(bz28833)$ git bz apply 28833

Bug 28833 - Speed up holds queue builder via parallel processing

126216 - Bug 28833: Speed up holds queue builder via parallel processing

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 28833: Speed up holds queue builder via parallel processing
Using index info to reconstruct a base tree...
M   C4/HoldsQueue.pm
M   debian/templates/koha-conf-site.xml.in
M   etc/koha-conf.xml
M   misc/cronjobs/holds/build_holds_queue.pl
Falling back to patching base and 3-way merge...
Auto-merging misc/cronjobs/holds/build_holds_queue.pl
CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl
Auto-merging etc/koha-conf.xml
CONFLICT (content): Merge conflict in etc/koha-conf.xml
Auto-merging debian/templates/koha-conf-site.xml.in
CONFLICT (content): Merge conflict in debian/templates/koha-conf-site.xml.in
Auto-merging C4/HoldsQueue.pm
CONFLICT (content): Merge conflict in C4/HoldsQueue.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel
processing

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-10-13 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #126202|0   |1
is obsolete||

--- Comment #22 from Kyle M Hall  ---
Created attachment 126216
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126216&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-10-13 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123996|0   |1
is obsolete||
 Attachment #126200|0   |1
is obsolete||

--- Comment #21 from Kyle M Hall  ---
Created attachment 126202
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126202&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-10-13 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #20 from Kyle M Hall  ---
Created attachment 126200
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126200&action=edit
Bug 28833: Balance the number of holds each chunk contains

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-20 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #19 from Kyle M Hall  ---
(In reply to David Cook from comment #17)
> Comment on attachment 123689 [details] [review]
> Bug 28833: Speed up holds queue builder via parallel processing
> 
> Review of attachment 123689 [details] [review]:
> -
> 
> ::: C4/HoldsQueue.pm
> @@ +210,5 @@
> > +
> > +DATA_LOOP:
> > +foreach my $chunk (@chunks) {
> > +my $pid = $pm->start and next DATA_LOOP;
> > +_ProcessBiblios($chunk);
> 
> I notice that the other _ProcessBiblios has arguments for $branches_to_use
> and $transport_cost_matrix. I'm guessing those should be included here too?

Good catch! I've fixed that issue and squashed the patches down for ease of
use.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-20 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123689|0   |1
is obsolete||
 Attachment #123785|0   |1
is obsolete||
 Attachment #123786|0   |1
is obsolete||

--- Comment #18 from Kyle M Hall  ---
Created attachment 123996
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123996&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #17 from David Cook  ---
Comment on attachment 123689
  --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689
Bug 28833: Speed up holds queue builder via parallel processing

Review of attachment 123689:
 --> 
(https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123689)
-

::: C4/HoldsQueue.pm
@@ +210,5 @@
> +
> +DATA_LOOP:
> +foreach my $chunk (@chunks) {
> +my $pid = $pm->start and next DATA_LOOP;
> +_ProcessBiblios($chunk);

I notice that the other _ProcessBiblios has arguments for $branches_to_use and
$transport_cost_matrix. I'm guessing those should be included here too?

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #16 from David Cook  ---
Comment on attachment 123786
  --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786
Bug 28833: (follow-up) Only spawn multiple processes if there are more records
than loops specified

Review of attachment 123786:
 --> 
(https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123786)
-

::: C4/HoldsQueue.pm
@@ +201,4 @@
>  # Split the list of bibs into groups to run in parallel
>  if ( $loops > 1 ) {
>  my $bibs_per_chunk = ceil( scalar @$bibs_with_pending_requests / 
> $loops );
> +$bibs_per_chunk = @$bibs_with_pending_requests if 
> @$bibs_with_pending_requests <= $bibs_per_chunk;

I don't understand this logic. 

scalar @$bibs_with_pending_requests = 10
$loops = 10
$bibs_per_chunk =  1

$bibs_per_chunk would still be 1.

Maybe I'm confused but I can't think of a case where $bibs_per_chunk could be
greater than scalar @$bibs_with_pending_requests.

--

I was thinking we'd do the $loops would be changed to $max_loops, and $loops
would be based on an arbitrary ratio. Perhaps 1000 bibs per 1 loop. 

So $loops = scalar @$bibs_with_pending_requests / 1000. 

If $loops < 1, then we don't fork. If $loops > 1, we double-check that we're
not exceeding our maximum number of loops. 

For example:
$max_loops = 4 
scalar @$bibs_with_pending_requests = 1
$loops = 10

We throttle $loops down from 10 to 4, because 4 is our maximum. 

Likewise, if @$bibs_with_pending_requests = 2000, we're only using 2 forks
instead of our maximum of 4, so we're not using more resources than necessary. 

--

Alternatively, we could have an arbitrary threshold of 1000. Below that and we
use $loops = 1. Above that, we use the $loops value supplied from config. 

I'm just trying to avoid scenarios where you have 50 Koha instances each
spinning up 10 child processes to process 2 requests each. 

But maybe I'm just prematurely optimizing. I suppose most instances will have
the default of 1 anyway.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #15 from Kyle M Hall  ---
Created attachment 123786
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786&action=edit
Bug 28833: (follow-up) Only spawn multiple processes if there are more records
than loops specified

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123783|0   |1
is obsolete||
 Attachment #123784|0   |1
is obsolete||

--- Comment #14 from Kyle M Hall  ---
Created attachment 123785
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123785&action=edit
Bug 28833: (follow-up) Move loops count control to koha conf

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #13 from Kyle M Hall  ---
Created attachment 123784
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123784&action=edit
Bug 28833: (follow-up) Only spawn multiple processes if there are more records
than loops specified

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-11 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #12 from Kyle M Hall  ---
Created attachment 123783
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123783&action=edit
Bug 28833: (follow-up) Move loops count control to koha conf

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #11 from David Cook  ---
The following is totally out of scope but inspired by the above:

What if default values in koha-conf.xml were determined based on a "profile"
provided for koha-create? 

By default, we'd have permissive values, so that inexperienced people could
have the most power possible through the Web UI, since they might not have a
sysadmin to update the backend. 

But we'd have a more restrictive profile that vendors and more experienced
people could use to have more locked down/secure Koha instances.

I'm trying to think of an approach that could make everyone happy...

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #10 from David Cook  ---
Another option might be to use koha-conf.xml instead of a system preference. 

I think it's important for it to be a configurable value per-instance, but I'm
opposed to the idea of library staff being able to determine server resource
usage.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

David Cook  changed:

   What|Removed |Added

 CC||dc...@prosentient.com.au

--- Comment #9 from David Cook  ---
This is an interesting one! 

On one hand, I like the use of a system preference to allow the value to be set
on a per-instance basis.

On the other hand, letting a library define an unrestricted number of forks for
a script seems potentially dangerous, especially if you have multiple Koha
instances on the same server.

I think this change needs more sanity checks.

Some options:
1) A CLI override that doesn't let the $loops be beyond X. This allows a
sysadmin to constrain the resource usage
2) Perhaps the size of $bibs_with_pending_requests should correlate with the
number of forks up to a maximum. (If you only have 10 bibs but your $loops is
10, it doesn't make sense to have 10 forks of 1 bib each. )
3) Maybe we should limit nproc in /etc/security/limits.d on a per-user basis
(but maybe this is best left to local sysadmins and this is a very blunt
instrument)

Actually, we should also do a int() check on $loops before feeding it to
Parallel::ForkManager, since it is user-provided input. (I suppose a non-number
would just cause a fatal error on a $loops > 1 check, but better to not fork
than to fail.)

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123688|0   |1
is obsolete||

--- Comment #8 from Kyle M Hall  ---
Created attachment 123689
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123679|0   |1
is obsolete||

--- Comment #7 from Kyle M Hall  ---
Created attachment 123688
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123688&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123671|0   |1
is obsolete||

--- Comment #6 from Kyle M Hall  ---
Created attachment 123679
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123679&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
0) Install the Perl library Parallel::Loops
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123659|0   |1
is obsolete||

--- Comment #5 from Kyle M Hall  ---
Created attachment 123671
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123671&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

Test Plan:
1) Generate a huge number of holds ( a few thousand at the minimum )
2) Run the holds queue builder, use the `time` utility to track how much
   time it took to run
3) Set HoldsQueueParallelLoopsCount to 10
4) Repeat step 2, note the improvement in speed
5) Experiment with other values for HoldsQueueParallelLoopsCount
6) prove t/db_dependent/HoldsQueue.t

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-10 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Status|NEW |Needs Signoff

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Jessie Zairo  changed:

   What|Removed |Added

 CC||jza...@bywatersolutions.com

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123657|0   |1
is obsolete||

--- Comment #4 from Kyle M Hall  ---
Created attachment 123659
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123659&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

 Attachment #123656|0   |1
is obsolete||

--- Comment #3 from Kyle M Hall  ---
Created attachment 123657
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123657&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #2 from Kyle M Hall  ---
This patch provides a tremendous speedup for large numbers of holds. However,
it causes HoldsQueue.t to fail spectacularly.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #1 from Kyle M Hall  ---
Created attachment 123656
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123656&action=edit
Bug 28833: Speed up holds queue builder via parallel processing

The holds queue builder can take a very long time to run on systems with many
holds. For example, a partner with 124,784 unfilled ( not found ) holds, is
taking about 64 minutes to run. If we run that same number of holds in 5
parallel chunks ( splitting the number of records as evenly as possible, but
*not* taking into account the holds per bib ), it takes 21.5 minutes. If we use
10 loops, it takes less then 14 minutes.

-- 
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

2021-08-09 Thread bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

Kyle M Hall  changed:

   What|Removed |Added

   Assignee|koha-b...@lists.koha-commun |k...@bywatersolutions.com
   |ity.org |

-- 
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/