[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Lucas Gass changed: What|Removed |Added Status|Pushed to master|Pushed to stable Version(s)|22.11.00|22.11.00, 22.05.03 released in|| --- Comment #16 from Lucas Gass --- Backported to 22.05.x for 22.05.03 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Severity|enhancement |normal --- Comment #15 from Nick Clemens --- Performance bugs should be considered bug, not enhancement, remarking. Please consider for backport -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Lucas Gass changed: What|Removed |Added CC||lu...@bywatersolutions.com --- Comment #14 from Lucas Gass --- Enhancement will not be backported to 22.05.x -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #13 from Tomás Cohen Arazi --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Tomás Cohen Arazi changed: What|Removed |Added Version(s)||22.11.00 released in|| Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when importing records with many items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Summary|Improve performance of |Improve performance of |AddBiblio when record has |AddBiblio when importing |items |records with many items -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when record has items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #12 from Marcel de Rooy --- (In reply to Nick Clemens from comment #11) > > That's exactly the issue - we aren't adding items, but we are processing > item fields - what would you suggest for the bug title? Well, lets finish this here. What about: Improve performance of AddBiblio when importing records with many items ? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when record has items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #11 from Nick Clemens --- (In reply to Marcel de Rooy from comment #10) > (In reply to Nick Clemens from comment #9) > > (In reply to Marcel de Rooy from comment #8) > > > I agree with this patch, no problem about that. > > > But the performance gain should be close to zero. > > > Running Biblio.t (calling AddBIblio) twice with and without patch: > > > RM: I would suggest to revise the title unless we have clear feedback to > > > prove the statement. Keeping the release notes a bit more realistic? > > > > The gain is only going to come when you have records that contain items. > > > > Testing with NYTProf against an import with record containing items, there > > was a clear improvement > > > > Biblio.t does not add items to most records, so the savings will be minimal > > if present > > When you look at the AddBiblio code, you will be surprised to see that this > routine does not add items at all. > During import you need something as BatchCommitItems to add items. > > So the title of this report still needs attention :) That's exactly the issue - we aren't adding items, but we are processing item fields - what would you suggest for the bug title? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when record has items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #10 from Marcel de Rooy --- (In reply to Nick Clemens from comment #9) > (In reply to Marcel de Rooy from comment #8) > > I agree with this patch, no problem about that. > > But the performance gain should be close to zero. > > Running Biblio.t (calling AddBIblio) twice with and without patch: > > RM: I would suggest to revise the title unless we have clear feedback to > > prove the statement. Keeping the release notes a bit more realistic? > > The gain is only going to come when you have records that contain items. > > Testing with NYTProf against an import with record containing items, there > was a clear improvement > > Biblio.t does not add items to most records, so the savings will be minimal > if present When you look at the AddBiblio code, you will be surprised to see that this routine does not add items at all. During import you need something as BatchCommitItems to add items. So the title of this report still needs attention :) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio when record has items
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Summary|Improve performance of |Improve performance of |AddBiblio |AddBiblio when record has ||items --- Comment #9 from Nick Clemens --- (In reply to Marcel de Rooy from comment #8) > I agree with this patch, no problem about that. > But the performance gain should be close to zero. > Running Biblio.t (calling AddBIblio) twice with and without patch: > RM: I would suggest to revise the title unless we have clear feedback to > prove the statement. Keeping the release notes a bit more realistic? The gain is only going to come when you have records that contain items. Testing with NYTProf against an import with record containing items, there was a clear improvement Biblio.t does not add items to most records, so the savings will be minimal if present -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Marcel de Rooy changed: What|Removed |Added QA Contact|testo...@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Marcel de Rooy changed: What|Removed |Added Status|BLOCKED |Passed QA --- Comment #8 from Marcel de Rooy --- I agree with this patch, no problem about that. But the performance gain should be close to zero. Running Biblio.t (calling AddBIblio) twice with and without patch: WITH PATCH real0m3.228s user0m2.505s sys 0m0.371s real0m2.786s user0m2.176s sys 0m0.370s WITHOUT PATCH real0m3.162s user0m2.456s sys 0m0.374s real0m2.728s user0m2.104s sys 0m0.403s The funny thing here is that it was faster without the patch. Which does not tell me anything. Just incidental. If we cached the marc structure, the other calls is pure memory stuff on a few fields. This really costs nothing. The title of this report promises way too much. No gain expected. RM: I would suggest to revise the title unless we have clear feedback to prove the statement. Keeping the release notes a bit more realistic? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Marcel de Rooy changed: What|Removed |Added CC||m.de.r...@rijksmuseum.nl Status|Passed QA |BLOCKED --- Comment #7 from Marcel de Rooy --- Taking a short look here -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Martin Renvoize changed: What|Removed |Added Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Martin Renvoize changed: What|Removed |Added CC||martin.renvoize@ptfs-europe ||.com Status|Needs Signoff |Signed Off --- Comment #6 from Martin Renvoize --- Trivial and clear code improvement.. we don't handle item table rows as part of AddBiblio so there's really no reason to process them. Unit tests continue to all pass as expected, no regressions found and the QA script is happy. I can confirm the performance improvement too alongside the cleaner feeling code. Think I'm going to go straight to QA for this one. Passed. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Martin Renvoize changed: What|Removed |Added Attachment #135216|0 |1 is obsolete|| --- Comment #5 from Martin Renvoize --- Created attachment 135238 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135238&action=edit Bug 30789: Improve performance of AddBiblio AddBiblio calls TransformMarcRecordToKoha. We are only interested in getting biblio and biblioitems info, so we can pass 'no_items' to save some field lookups Benchmarking saw a ~75% increase in the performance of this call To test: 1 - Stage and import some records 2 - Confirm it works the same before and after patch 3 - Add a biblio via cataloguing, confirm it works Signed-off-by: Martin Renvoize -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 David Cook changed: What|Removed |Added CC||dc...@prosentient.com.au --- Comment #4 from David Cook --- Adding this one to my to test list as I'd love this... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Blocks||30813 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30813 [Bug 30813] Refactor TransformMarcToKoha to remove TransformMarcToKohaOneField -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Attachment #135071|0 |1 is obsolete|| --- Comment #3 from Nick Clemens --- Created attachment 135216 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135216&action=edit Bug 30789: Improve performance of AddBiblio AddBiblio calls TransformMarcRecordToKoha. We are only interested in getting biblio and biblioitems info, so we can pass 'no_items' to save some field lookups Benchmarking saw a ~75% increase in the performance of this call To test: 1 - Stage and import some records 2 - Confirm it works the same before and after patch 3 - Add a biblio via cataloguing, confirm it works -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #2 from Jonathan Druart --- I wouldn't pass the mss, we are already retrieving it from cache using unsafe. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added CC||jonathan.druart+koha@gmail. ||com Assignee|koha-b...@lists.koha-commun |n...@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 --- Comment #1 from Nick Clemens --- Created attachment 135071 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135071&action=edit Bug 30789: Improve performance of AddBiblio AddBiblio calls TransformMarcRecordToKoha. We are only interested in getting biblio and biblioitems info, so we can pass 'no_items' to save some field lookups Additionally we can pass the subfield structure we have already fetched into the subroutine TransformMarcToKohaOneField Testing with NYTProf I saw a reduction of ~6 seconds when importing ~400 bibs and ~1000 items i.e. I exported and imported the sample data (after removing duplicate 010 from record 72) To test: 1 - Stage and import some records 2 - Confirm it works the same before and after patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 30789] Improve performance of AddBiblio
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30789 Nick Clemens changed: What|Removed |Added Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
