[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Wainui Witika-Park changed: What|Removed |Added Version(s)|24.11.00,24.05.06 |24.11.00 released in|| CC||[email protected] ||et.nz --- Comment #30 from Wainui Witika-Park --- (In reply to Lucas Gass (lukeg) from comment #28) > Backported to 24.05.x for upcoming 24.05.06 Reverted from 24.05 as it was causing tests to fail on jenkins -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Fridolin Somers changed: What|Removed |Added CC||[email protected] ||m Status|Pushed to oldstable |Needs documenting --- Comment #29 from Fridolin Somers --- This script changed a lot since Bug 29440 so this does not apply easy on 23.11.x I propose to not backport for stability -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Lucas Gass (lukeg) changed: What|Removed |Added Status|Pushed to stable|Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Lucas Gass (lukeg) changed: What|Removed |Added CC||[email protected] Version(s)|24.11.00|24.11.00,24.05.06 released in|| Status|Pushed to main |Pushed to stable --- Comment #28 from Lucas Gass (lukeg) --- Backported to 24.05.x for upcoming 24.05.06 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Katrin Fischer changed: What|Removed |Added Version(s)||24.11.00 released in|| Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #27 from Katrin Fischer --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Katrin Fischer changed: What|Removed |Added Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #26 from Mason James --- (In reply to Katrin Fischer from comment #25) > We have a new dependeny here - we need Mason to give his OK first. great news, libmarclint-perl package is available in debian repo so, all ok :) https://tracker.debian.org/pkg/libmarc-lint-perl -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Katrin Fischer changed: What|Removed |Added Status|Passed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #25 from Katrin Fischer --- We have a new dependeny here - we need Mason to give his OK first. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added QA Contact|[email protected] |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added CC||bernard.scaife@ptfs-europe. ||com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added Attachment #173833|0 |1 is obsolete|| --- Comment #24 from Martin Renvoize (ashimema) --- Created attachment 173926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173926&action=edit Bug 37478: Rename strict to skip_bad_records and add POD Option name changed, POD updated. The reason not to do this by default is that it is an extra record conversion to check the validity, so may slow down the import. Signed-off-by: Martin Renvoize -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added Attachment #173832|0 |1 is obsolete|| --- Comment #23 from Martin Renvoize (ashimema) --- Created attachment 173925 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173925&action=edit Bug 37478: (follow-up) Add MARC::Lint to cpanfile Signed-off-by: David Nind Signed-off-by: Martin Renvoize -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Martin Renvoize (ashimema) changed: What|Removed |Added Attachment #173831|0 |1 is obsolete|| --- Comment #22 from Martin Renvoize (ashimema) --- Created attachment 173924 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173924&action=edit Bug 37478: Add strict mode to bulkmarcimport This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors. If errors are found, we then lint the record to catch any problems, output the warnings, and skip the record. To test: 1 - Download the sample records from this bug report 2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v fails! 3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v fails! 4 - Apply patch 5 - Repeat 2 & 3 - no change 6 - Add -st switch to the commands: perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st 7 - The records are now skipped, and the script completes 8 - Confirm the warnings generated are useful 9 - Sign off! Signed-off-by: David Nind Signed-off-by: Martin Renvoize -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Text to go in the|This adds a -st strict |This adds a -sk release notes|option to |(--skip_bad_records) option |misc/migration_tools/bulkma |to |rcimport.pl. This can be|misc/migration_tools/bulkma |used to catch any parsing |rcimport.pl. Use this |errors - if errors are |option to catch any parsing |found, the record is|errors - if errors are |checked to identify any |found, the record is |problems, outputs warnings, |checked to identify any |and then skips the record. |problems, outputs warnings, ||and then skips the record. ||If this option is not used, ||and there are bad records, ||the import job may fail. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #20 from Nick Clemens (kidclamp) --- Created attachment 173832 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173832&action=edit Bug 37478: (follow-up) Add MARC::Lint to cpanfile Signed-off-by: David Nind -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #21 from Nick Clemens (kidclamp) --- Created attachment 173833 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173833&action=edit Bug 37478: Rename strict to skip_bad_records and add POD Option name changed, POD updated. The reason not to do this by default is that it is an extra record conversion to check the validity, so may slow down the import. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added Attachment #170703|0 |1 is obsolete|| Attachment #170704|0 |1 is obsolete|| --- Comment #19 from Nick Clemens (kidclamp) --- Created attachment 173831 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173831&action=edit Bug 37478: Add strict mode to bulkmarcimport This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors. If errors are found, we then lint the record to catch any problems, output the warnings, and skip the record. To test: 1 - Download the sample records from this bug report 2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v fails! 3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v fails! 4 - Apply patch 5 - Repeat 2 & 3 - no change 6 - Add -st switch to the commands: perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st 7 - The records are now skipped, and the script completes 8 - Confirm the warnings generated are useful 9 - Sign off! Signed-off-by: David Nind Current status: Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #18 from Jonathan Druart --- Yes but then --strict is confusing IMO, would not --skip-bad-records (or something similar) be more appropriate? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #17 from David Nind --- My understanding (limited as it is) is that it skips "bad" records if you use the new --strict option, instead of dying/failing. Which I guess means that your import will work, except for the "bad" records. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #16 from Jonathan Druart --- (In reply to Jonathan Druart from comment #15) > Does it mean that the record is imported if there are errors? > > If so shouldn't the strict more be the default? Ha, it dies, sorry. Then isn't it less... strict? I feel like I am missing something, just ignore me if I am not making sense. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #15 from Jonathan Druart --- Does it mean that the record is imported if there are errors? If so shouldn't the strict more be the default? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Text to go in the|This adds a -st strict |This adds a -st strict release notes|option to |option to |misc/migration_tools/bulkma |misc/migration_tools/bulkma |rcimport.pl. This can be|rcimport.pl. This can be |used to catch any parsing |used to catch any parsing |errors - if errors are |errors - if errors are |found, the record is|found, the record is |checked to catch any|checked to identify any |problems, outputs the |problems, outputs warnings, |warnings, and skips the |and then skips the record. |record. | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Text to go in the||This adds a -st strict release notes||option to ||misc/migration_tools/bulkma ||rcimport.pl. This can be ||used to catch any parsing ||errors - if errors are ||found, the record is ||checked to catch any ||problems, outputs the ||warnings, and skips the ||record. --- Comment #14 from David Nind --- Testing notes (using KTD): 1. After applying the patch, and to get the new Perl module to install, I: 1.1 Shutdown KTD 1.2 Edited my .env file so that: INSTALL_MISSING_FROM_CPANFILE=yes 1.3 Restarted KTD -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Attachment #170601|0 |1 is obsolete|| --- Comment #13 from David Nind --- Created attachment 170704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170704&action=edit Bug 37478: (follow-up) Add MARC::Lint to cpanfile Signed-off-by: David Nind -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Attachment #169598|0 |1 is obsolete|| --- Comment #12 from David Nind --- Created attachment 170703 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170703&action=edit Bug 37478: Add strict mode to bulkmarcimport This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors. If errors are found, we then lint the record to catch any problems, output the warnings, and skip the record. To test: 1 - Download the sample records from this bug report 2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v fails! 3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v fails! 4 - Apply patch 5 - Repeat 2 & 3 - no change 6 - Add -st switch to the commands: perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st 7 - The records are now skipped, and the script completes 8 - Confirm the warnings generated are useful 9 - Sign off! Signed-off-by: David Nind -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=37709, ||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=35466, ||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=37564, ||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=37550 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #11 from Nick Clemens (kidclamp) --- Created attachment 170601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170601&action=edit Bug 37478: (follow-up) Add MARC::Lint to cpanfile -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Katrin Fischer changed: What|Removed |Added CC||[email protected] --- Comment #10 from Katrin Fischer --- Adding Mason: https://wiki.koha-community.org/wiki/Release_management#Dependency_changes -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Cook changed: What|Removed |Added Keywords||dependency Status|Needs Signoff |Failed QA --- Comment #9 from David Cook --- (In reply to David Nind from comment #8) > Getting this error after applying the patch for step 5 - assuming this means > the Perl MARC::Lint module is not installed and needs adding (however that > is done): Bingo! Marking Failed QA as we'll need it on the cpanfile at least I believe. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 David Nind changed: What|Removed |Added CC||[email protected] --- Comment #8 from David Nind --- Getting this error after applying the patch for step 5 - assuming this means the Perl MARC::Lint module is not installed and needs adding (however that is done): perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v Can't locate MARC/Lint.pm in @INC (you may need to install the MARC::Lint module) (@INC contains: /kohadevbox/koha /kohadevbox/koha/lib /kohadevbox/qa-test-tools /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at misc/migration_tools/bulkmarcimport.pl line 12. BEGIN failed--compilation aborted at misc/migration_tools/bulkmarcimport.pl line 12. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added Assignee|[email protected] |[email protected] |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #7 from David Cook --- (In reply to Nick Clemens (kidclamp) from comment #0) > Two cases we have seen are an invalid characters: > :9: parser error : PCDATA invalid Char value 31 > aPGcC > > and invalid/missing subfield: > :91: parser error : attributes construct error > Among nineteenth-century women's rights reformers, > Elizab Of course, parser errors are often how we find out that we have bad metadata... oh well. I'll raise the issues anyway. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #6 from David Cook --- (In reply to David Cook from comment #5) > So yeah I'd certainly say a bug in MARC::File::XML. I'll raise 2 issues in Galen's marc-perl git repo for the attributes issue and the invalid characters. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #5 from David Cook --- (In reply to Nick Clemens (kidclamp) from comment #0) > It turns out MARC::Record can read the incoming file - we can convert it to > xml to save as metadata, but only on trying to create the record out of the > xml do we die. > > This may point to a bug in MARC::Record This sounds very familiar. I remember talking to someone (maybe even you heh) or commenting on a bug somewhere about invalid XML being produced by MARC::Record... I recall that MARC::File::XML has some shockingly bad code when it comes to creating XML. If you look at the source code on MetaCPAN, you'll see that it actually recreates the record as a string, and it only escapes &, <, and >. So yeah I'd certainly say a bug in MARC::File::XML. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #4 from Nick Clemens (kidclamp) --- The reason to not simply use Lint here was a matter of how sensitive it is. Lint doesn't distinguish between an error that breaks parsing vs a marc rules error like wrong indicator value or repeated field that should be singular. We worried it would be too strict -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added CC||[email protected], ||[email protected], ||[email protected], ||martin.renvoize@ptfs-europe ||.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #3 from Nick Clemens (kidclamp) --- Created attachment 169600 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=169600&action=edit File with a missing subfiled indicator -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #2 from Nick Clemens (kidclamp) --- Created attachment 169599 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=169599&action=edit File with invalid character in 003 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 --- Comment #1 from Nick Clemens (kidclamp) --- Created attachment 169598 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=169598&action=edit Bug 37478: Add strict mode to bulkmarcimport This patch adds a conversion from MARC -> XML -> MARC to catch any parsing errors. If errors are found, we then lint the record to catch any problems, output the warnings, and skip the record. To test: 1 - Download the sample records from this bug report 2 - perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v fails! 3 - perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v fails! 4 - Apply patch 5 - Repeat 2 & 3 - no change 6 - Add -st switch to the commands: perl misc/migration_tools/bulkmarcimport.pl -b --file=520_nosubfield.mrc -v -st perl misc/migration_tools/bulkmarcimport.pl -b --file=003_subfielda.mrc -v -st 7 - The records are now skipped, and the script completes 8 - Confirm the warnings generated are useful 9 - Sign off! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 37478] bulkmarcimport.pl can die on bad records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37478 Nick Clemens (kidclamp) changed: What|Removed |Added Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Koha-bugs mailing list [email protected] https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
