[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Phil Ringnalda changed: What|Removed |Added CC||p...@chetcolibrary.org -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Katrin Fischer changed: What|Removed |Added CC||katrin.fisc...@bsz-bw.de --- Comment #19 from Katrin Fischer --- (In reply to Héctor Eduardo Castro Avalos from comment #18) > Hi everyone > > This is a problematic subject, keep-up to date the frameworks. We fix this > updating SQL manually. Right now I encourage all my co-worker to start > cataloging in Rancor, rather than frameworks. You'll see that this is a > better option with the unique inconvenience that you cannot obtain > stadistics with frameworks anymore. > > Of course Rancor got some problems (bugs) yet, but if you learn how to avoid > them, you'll see a good option for cataloging process. And after all it's > much faster cataloging there than the old frameworks. > > We used frameworks with students and inexperienced librarians. > > Cataloger thought. Hector, just a note: Even when using Rancor, always make sure that you define fields/subfields used in the default framework. It will cause you data loss if the record is opened in the old editor otherwise. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Héctor Eduardo Castro Avalos changed: What|Removed |Added CC||hector.hecax...@gmail.com --- Comment #18 from Héctor Eduardo Castro Avalos --- Hi everyone This is a problematic subject, keep-up to date the frameworks. We fix this updating SQL manually. Right now I encourage all my co-worker to start cataloging in Rancor, rather than frameworks. You'll see that this is a better option with the unique inconvenience that you cannot obtain stadistics with frameworks anymore. Of course Rancor got some problems (bugs) yet, but if you learn how to avoid them, you'll see a good option for cataloging process. And after all it's much faster cataloging there than the old frameworks. We used frameworks with students and inexperienced librarians. Cataloger thought. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 emj...@pa.net changed: What|Removed |Added CC||emj...@pa.net --- Comment #17 from emj...@pa.net --- Re: David Cook: "I'm thinking that there should be some mechanism for updating an existing framework." Koha requires that the "Default" Framework must remain, unedited and available, in Koha. This is the area to do all MARC21 updates, hopefully with each update of Koha. As David Cook noted, a library may set up a unique Framework that does not have all of the MARC fields and subfields when such a library is specifically attempting to keep things simplified and efficient. I am the only cataloger among seven people who add bibliographic records to our system. I ran reports to see which MARC fields were used, checked the ones that were used only a handful of times (to see if they were used correctly), and changed all of our records back to the Koha Default Framework. I deleted all the other frameworks (as none of their fixed fields were coded correctly) and painstakingly created a "new" default framework, deleting fields and subfields that are obsolete or not relevant to the work that we do. The most important thing to me, regarding MARC updates, is that I do not have to start from scratch using the Koha Default Framework. As long as the information is added to the Koha Default Framework, I can manually look it up and manually add it (correctly) to any of my unique Frameworks. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Marjorie Barry-Vila changed: What|Removed |Added CC||marjorie.barry-vila@collect ||o.ca -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #16 from David Cook --- Sadly...it appears that fixing the cataloguing editor, so that it doesn't discard fields/subfields not present in the framework, would require re-writing at least the "build_tabs" sub in addbiblio.pl (ie the cataloguing editor). Since that there is a core bit of Koha functionality, I'm not planning to touch it. Half-tempted to turn "build_tabs" into a syspref controlled wrapper that can switch between the current "build_tabs" code and a new sub. On one hand, that would keep the traditional interface untouched, so no regressions would happen. On the other hand, it would mean maintaining two slightly different versions of the same editor, and then trying to merge those together over time. Unfortunately, we already have that problem with the QueryParser :/. When a syspref or fallbacks are in place, there's no incentive to improve the new development... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #15 from David Cook --- tl;dr The top priority should be making sure that no unwanted data loss occurs. One solution might be to make sure that incoming fields/subfields are saved, even if they don't appear in a framework. (I suppose this solution has flaws in terms of determining labels and repeatability... I suppose you could say "unknown field/subfield" and default to being repeatable in the editor.) (For cataloguers who count on the workflow of fields/subfields not being saved if they're not in the framework, we could have a "strict" flag at the framework level which could control whether or not "unknown subfields/fields" are saved.) This solution would actually negate some of the need to update frameworks. Updates to labels and whether it's "repeatable" would be the only thing needed really, as the frameworks would be acting more as templates than as strict frameworks. Thoughts? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477
--- Comment #14 from David Cook ---
I'm thinking more about how to keep frameworks current...
In theory, all MARC Bibliographic Frameworks should include all MARC
Bibliographic fields and subfields.
I can only think of two scenarios where this theory would be false:
1) You've bent the MARC Bibliographic Framework in an attempt to support a
different sort of metadata format.
[Note: I don't think this is an overly valid reason not to auto-update
frameworks.]
2) You want a framework that will discard certain fields when importing
records. For instance, you might never ever want to keep the 035 for incoming
records (although it's a nice bit of data to keep). So you delete the 035 from
the framework.
[Note: This scenario is more interesting. By deleting certain fields, you could
potentially increase the efficiency of your cataloguing workflow. That is, you
would reduce the amount of time spent deleting unwanted data when importing
records.
Yet... I can't help but feel that the "xkcd: Workflow" comic
(http://xkcd.com/1172/) is relevant here.
I suppose a way to mitigate this might be to add an "auto-update" flag to the
frameworks. You could turn it off for frameworks whose structure you don't want
to change.
Yet...that doesn't make sense either...because there will come a time where you
need to add fields/subfields (such as the RDA fields/subfields).
I suppose that leads us back to doing manual updates... but that can be painful
for a number of reasons. One of those being that the full list of missing
fields/subfields will show up each time, so you need to hand-pick which
fields/subfields you want to add.
Plus, if you're doing manual updates, you need a prompt to do the updates. I
suppose this could be done with a staff client news item...but then when to do
framework update checks...]
---
I've been thinking more about visibility. Regardless of what metadata format we
use, we'll run into this problem again and again.
Instead of using one column ("hidden"), it might make sense to use three
columns ("internal","external","editor").
"internal" and "external" would control visibility in the staff client and the
OPAC (maybe OAI as well...hence my use of "external" as a label).
"editor" would have more options: "used","used - collapsed", "hidden",
"unused". In this case, "used" shows up uncollapsed, "used - collapsed" shows
up as a collapsed field, "hidden" doesn't show up unless you're importing
records with a value in that subfield, "unused" means that there can never be a
value in this field.
Alternatively... we keep the current "hidden" column and add a "used" column
which defaults to "on". If someone wants a framework that will ignore certain
fields when importing, they can turn "used" to "off".
In support of this mechanism, it might make sense to remove the ability for
people to delete fields/subfields from frameworks (unless they're local
fields/subfields that they've added). Of course, that's easy to circumvent by
just importing a framework that doesn't have certain fields/subfields (until
the next auto-update comes around which re-populates those missing
fields/subfields).
---
I don't know... I've looked at this a bit in DSpace as well, and I don't really
like the way they do it either. In DSpace, there is a metadata registry, then
you define input forms (which would be analogous to our frameworks in a way).
It gets incredibly unwieldy after the first form or two, especially if you have
a lot of fields per form.
In the past, I've mostly done original cataloguing, so I'm not 100% sure what
other ILSes/LMSes do for templates with copy cataloguing. I recall cataloguers
using Horizon would have to delete unwanted data when importing records. I
think the function of the templates were more so to provide a minimal set of
fields/subfields that you should fill out. (You then had the option to add more
fields/subfields manually as necessary.)
---
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #13 from David Cook --- Created attachment 29961 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29961&action=edit DBIC-based framework updater Here is some more complex code that I've devised for updating frameworks. You give it an arrayref of hashrefs, a DBIC resultset, and an arrayref of framework codes. The script will check to see if a row matching the hashref (using the primary key specified in the resultset) exists for each framework. If it exists, it runs an optional "update" callback. If it doesn't exist, it runs an optional "insert" callback (or just inserts it into the database if no callback has been specified). Prior to the database search, the "canon_callback" can be used to modify the hashref for each frameworkcode. In my use case, if I'm loading marc_subfield_structure, I want to change the hidden value to -6 as I'm updating existing frameworks. Of course, if we were loading marc_tag_structure or were installing a framework for the first time, we wouldn't want to do that. I also use the insert/update callbacks to build messages that I sent to the template UI so that the user knows what has been updated. It's not very fast, but DBIC itself says "DBIx::Class is not built for speed, it's built for convenience and ease of use". It certainly is convenient and easy to use :). I have debated the utility of this class a bit. I figure it could be handy when we're installing the sample frameworks, as we could use the "canon_callback" to change the "hidden" value on a per framework and subfield basis. We wouldn't even need the insert/update callbacks. [Note: The DBIx::Class docs do mention that using find_or_create (and thus find_or_new or update_or_create) is subject to a race condition and that it should be run within a transaction. Mind you, lots of what we do in Koha could be subject to a race condition, so I don't know how important it is to get this right at the moment. In any case, there seems to be a few ways of using transactions with DBIC (http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/Manual/Cookbook.pod#TRANSACTIONS). It's not really a big deal when initially installing frameworks, as there shouldn't be anyone to race against, but worth mentioning for updates and in general.] -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477
--- Comment #12 from David Cook ---
As for using DBIx::Class, it's literally this simple:
use Koha::Database;
use XML::Simple;
my $schema = Koha::Database->new()->schema();
my $SimpleXML = new XML::Simple;
my $field_file = "../marc21_tag.xml";
my $canonical_field_structure = '';
if ( -e $field_file){
my $canonical_field_structure = $SimpleXML->XMLin($field_file, KeyAttr =>
[]);
foreach my $structure
(@{$canonical_field_structure->{marc_tag_structure}}){
$schema->resultset("MarcTagStructure")->find_or_create($structure);
}
}
Of course, this is just an example. I'd prefer to use XML::LibXML rather than
XML::Simple (especially as XML::Simple itself states that new code should avoid
using XML::Simple), and set up the logical structure a bit differently, but
yeah...that's it.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #11 from David Cook --- (In reply to Bernardo Gonzalez Kriegel from comment #10) > I created a new fw, set hidden = -6 for all subtags. A new record using this > fw will only show those tags/subtags that are mandatory. > Editing an existing record will show any tag/subtag that have a value, even > hidden ones, and on saving no information is loose. Ah, my mistake! That's good! I'm glad that I was wrong! It appears that this is also the case when using Z39.50. I think the problem I may have encountered in the past was losing data when importing a record that contained fields/subfields that didn't exist in that framework. *checks with an example* Aha! Yes! That's what it is! It makes sense...if the field/subfield doesn't exist in the framework, it has no way of including it in the record. I suppose this might be both a pro and a con... When copy-cataloguing, you might only want to import a certain subset of fields, so you'd want to ignore fields not defined in that framework. On the other hand, if you're cataloguing RDA records and you're not familiar with the extra fields and subfields needed for RDA, you'll lose data that you might want to keep when importing. That said, I think the onus needs to be on librarians to properly configure their frameworks for that. I don't know if this distinction is necessarily apparent in Koha. I imagine it's possible that a librarian could create a new framework with a subset of fields, and lose data when importing records because they forgot a field/subfield. But...you could argue the onus is still on the person configuring the framework. If they didn't include the field/subfield, they have no one but themselves to blame for lost data. > I think that default and sample (or simple) frameworks needs only 3 hidden > values: visible (0), editable but not visible (2), and hidden (5) > [visibility on opac/staff]. (5) could be used for obsoleted/deprecated > tags/subtags (e.g. 440), but again you can't hide something in the editor > with a value > > (-6) is the value I was told to use when adding new tags/subtags, they do > not display on editor (for new) and produce no functional change, but > imported record can show values [opac/staff]. That sounds reasonable to me. > Whichever we use, my thinking is that we need to use a few hidden values as > possible, and default FW needs to show as many tags as possible Agreed. > Then sample FW could be built by copying default, and _hiding_ (editor) all > but a selected subset of tags. Which subset... I don't care, but it need a > certain rationale behind. Not an universal agreement though. That also makes sense. I don't know which subset either. While there are cataloguing standards, I find practice is often highly localized, so they'll need to be re-configured by end-users anyway. It might be useful to get input from cataloguers that work with a range of materials for this one. I mostly catalogued books, so I would say Leader,005,008,100(a),245(a,b,c),260(a,b,c) (maybe also 264(a,b,c) as it's used in RDA),300(a),336(a,2),337(a,2),338(a,2),490(a,v,x),500(a),505(a,r,t),520(a,b),700(a,t,d,f). I don't have a lot of experience with serials but it should probably include 773(a,t,d,g,w). Other frameworks would probably benefit from cataloguers with more specialized knowledge and experience. > But if they have all tags/subs then no information will be loose. > That's what we need to ensure Agreed. In that case, what do you think about the Fast Add Framework? It only includes a small collection of fields/subfields. Wouldn't it make more sense to include all the fields, and only show the subset that it currently shows? While I'm somewhat skeptical of the utility of many of the sample frameworks, I think the Fast Add Framework is quite useful. Admittedly, I don't know if it needs to have all the fields, as I doubt one would ever really be importing records into it. I suppose it would be good to have all the fields in case you accidentally switched to the Fast Add Framework and saved a record. That would lose data... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #10 from Bernardo Gonzalez Kriegel --- (In reply to Jacek Ablewicz from comment #9) > (In reply to David Cook from comment #8) > > > If a field doesn't appear in the Editor, and it gets re-saved...any fields > > not appearing in the Editor will be destroyed. > > Does it really work that way for biblos MARC frameworks?? If so, that would > be a _major_ disaster IMO. > I created a new fw, set hidden = -6 for all subtags. A new record using this fw will only show those tags/subtags that are mandatory. Editing an existing record will show any tag/subtag that have a value, even hidden ones, and on saving no information is loose. I think that default and sample (or simple) frameworks needs only 3 hidden values: visible (0), editable but not visible (2), and hidden (5) [visibility on opac/staff]. (5) could be used for obsoleted/deprecated tags/subtags (e.g. 440), but again you can't hide something in the editor with a value (-6) is the value I was told to use when adding new tags/subtags, they do not display on editor (for new) and produce no functional change, but imported record can show values [opac/staff]. Whichever we use, my thinking is that we need to use a few hidden values as possible, and default FW needs to show as many tags as possible Then sample FW could be built by copying default, and _hiding_ (editor) all but a selected subset of tags. Which subset... I don't care, but it need a certain rationale behind. Not an universal agreement though. But if they have all tags/subs then no information will be loose. That's what we need to ensure -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Jacek Ablewicz changed: What|Removed |Added CC||a...@biblos.pk.edu.pl --- Comment #9 from Jacek Ablewicz --- (In reply to David Cook from comment #8) > If a field doesn't appear in the Editor, and it gets re-saved...any fields > not appearing in the Editor will be destroyed. Does it really work that way for biblos MARC frameworks?? If so, that would be a _major_ disaster IMO. I was under the impression that if the certain subfields are set up as "hidden in editor", they still get saved (not getting destroyed on re-save). Also, at least in the manner of speaking, they do appear in editor ;), their contents are just not visible and not editable (AFAIRC authority editor uses "display: none" for that purpose, I'm not sure about biblio editor). > I figure...if it doesn't appear in the Editor, it probably _doesn't_ need to > appear anywhere else. There are some scenarios when we e.g. may want some fields/subfields to be: preserved in the MARC record (control numbers, some linking data for records imported from union catalogues etc.), visible in staff client, but not directly editable by the librarians. Possibility to define (perhaps even greatly) simplified frameworks, intended for some specific editing tasks (= with a lot of existing record fields/subfields hidden on edit) might often come in handy as well. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #8 from David Cook --- Just having a thought about "hidden" values... These are the "hidden" values: -9 => Future use -8 => Flag -7 => OPAC !Intranet !Editor Collapsed -6 => OPAC Intranet !Editor !Collapsed -5 => OPAC Intranet !Editor Collapsed -4 => OPAC !Intranet !Editor !Collapsed -3 => OPAC !Intranet Editor Collapsed -2 => OPAC !Intranet Editor !Collapsed -1 => OPAC Intranet Editor Collapsed 0 => OPAC Intranet Editor !Collapsed 1 => !OPAC Intranet Editor Collapsed 2 => !OPAC !Intranet Editor !Collapsed 3 => !OPAC !Intranet Editor Collapsed 4 => !OPAC Intranet Editor !Collapsed 5 => !OPAC !Intranet !Editor Collapsed 6 => !OPAC Intranet !Editor !Collapsed 7 => !OPAC Intranet !Editor Collapsed 8 => !OPAC !Intranet !Editor !Collapsed 9 => Future use • URL : if checked, the subfield is an URL, and can be clicked These are their frequencies in "marc21_subtag.xml". "hidden="-9" (0 hits) "hidden="-8" (0 hits) "hidden="-7" (0 hits) "hidden="-6" (3037 hits) "hidden="-5" (59 hits) "hidden="-4" (0 hits) "hidden="-3" (0 hits) "hidden="-2" (0 hits) "hidden="-1" (240 hits) "hidden="9" (0 hits) "hidden="8" (0 hits) "hidden="7" (3 hits) "hidden="6" (4 hits) "hidden="5" (379 hits) "hidden="4" (1 hit) "hidden="3" (0 hits) "hidden="2" (0 hits) "hidden="1" (11 hits) "hidden="0" (225 hits) Some of these don't make all that much sense. For instance, -5 is functionally equivalent to -6. They're both in the OPAC and Intranet, and neither is in the Editor. The Collapsed quality really only applies to the Editor...so we may as well make the subfields with -5 be -6 by default. 6 and 7 are also functionally equivalent...although they're a bit bizarre. I don't know why you'd ever need something to show in the Intranet but not in the OPAC or the Editor. If a field doesn't appear in the Editor, and it gets re-saved...any fields not appearing in the Editor will be destroyed. I figure...if it doesn't appear in the Editor, it probably _doesn't_ need to appear anywhere else. Conversely, it _does_ make sense that fields that appear in the Editor might not show up in the OPAC or the Intranet (as they might be system values or just not relevant for most views). Following that logic, I find -6 to be a bit silly. I'd prefer 5 or 8, which are functionally equivalent. Of course, such distinctions get a bit...pedantic, so I rather just go with 1 "hidden" value for a function. Perhaps "-6" is the best one to use, since it's the most common for fields that we want to hide. Anyway, these are just some thoughts... when it comes to updating frameworks, I'm going to use -6 for the sake of consistency. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477
--- Comment #7 from David Cook ---
(In reply to Bernardo Gonzalez Kriegel from comment #6)
> Created attachment 29869 [details]
> Loader for XML files
>
> This new attach has a variant of XML files, with one element to represent
> data types, and a simple script to load the values into Koha database.
>
> I tried $dbh->do() for each value, or a prepare() and then execute(), but is
> too slow. It builds a single string with all values to insert, similar as
> the SQL files for frameworks.
>
> In my tests the script takes a little more than three times as long as mysql
> client to load default framework (tags/subtags), but is a difference between
> ~0.7s and ~2.5s, perhaps acceptable...
As you said before, the speed shouldn't matter too much if it's only being run
during initial install and then during rare subsequent updates.
I've been playing today with your XML files, and have been working on a script
for updating existing frameworks using Koha::Database/DBIx::Class.
The "find_or_new" method is quite useful in that case, as it'll insert new rows
and it'll give you a chance to do further actions for rows that already exist
(using the primary key as a reference point).
At the moment, I have it set up to add missing fields/subfields, and update the
"repeatable" column for existing fields if the XML values are different.
I'm going to add an option for updating "libopac" and "liblibrarian" as well. I
figure some libraries use custom text so they might not necessarily want to
override it according to the MARC specification.
Anyway...
--
I can put together a demo DBIC loader for the default framework XML, but I'm
wondering now how we want to handle installing the other sample frameworks. I
took a look at bug 10963 and it looks really good. I like the idea you're going
with.
I'm thinking maybe something like this:
http://search.cpan.org/~ribasushi/DBIx-Class-0.08270/lib/DBIx/Class/ResultSet.pm#Multiple_queries
my $framework = $schema->resultset('MarcTagStructure')->search({
frameworkcode => '',
});
then...
$framework->create({ frameworkcode => 'SR' });
$framework->create({ frameworkcode => 'BKS' });
Then probably some updates to change the hidden values. I'm not sure where we
would want to store the data about which subfields to hide in which framework
though. I suppose we could hard-code it into the script, but it might be nicer
to keep that information in a config file.
Not 100% sure what the best way to update the hidden column will be...but I
think that might be the way to do it.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #6 from Bernardo Gonzalez Kriegel --- Created attachment 29869 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29869&action=edit Loader for XML files This new attach has a variant of XML files, with one element to represent data types, and a simple script to load the values into Koha database. I tried $dbh->do() for each value, or a prepare() and then execute(), but is too slow. It builds a single string with all values to insert, similar as the SQL files for frameworks. In my tests the script takes a little more than three times as long as mysql client to load default framework (tags/subtags), but is a difference between ~0.7s and ~2.5s, perhaps acceptable... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #5 from Bernardo Gonzalez Kriegel --- (In reply to David Cook from comment #3) > Sorry for my delayed response! Competing priorities... No problem :) This task will take some time > I'm happy to go with XML. I suppose my only reservations are about the > brittleness of XML and processing time involved in parsing a huge XML file. > I suppose the brittleness shouldn't be a factor...we could validate the XML > before committing it and parsing errors are very loud so there probably is > no danger there. As for processing time...it's not like installing or > updating frameworks would be a frequent operation so lengthy processing time > might not matter (if it is even a thing). I used XML::Simple to load and build sql files from the attached examples. It does not take to much time, and anyway is something to use at install time or when updating frameworks, not frequent tasks :) > > Now that I think about it... are "liblibrarian" and "libopac" the only > fields that need to be translated? Yes, and in EN and ES they are the same text. I wonder if there are libraries using different labels for opac/staff In french UNIMARC frameworks I found that libopac is shortened or empty in most cases. > marc_tag_structure: > 1) MARC: tagfield, liblibrarian, libopac, repeatable > 2) Koha: mandatory, authorised_value, frameworkcode In this case authorised_value... is not used, or can't be used. FW editor allows to set this to an authorised value, supposedly to list allowed values for indicators, but this is not used on editor (there is another bug that mentions this) > marc_subfield_structure: > 1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength > 2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode, > value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue In this case seealso is not used anymore (or that is implied by http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6007#c3) > > Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the > MARC specification data. Then we pass that through an XSLT which adds the > relevant "Koha" data. > > Think about it...in most cases those "Koha" fields are "null" or "". The > most important fields are probably "tab" and "hidden". If we use defaults > for most cases, but define exceptions in the XSLT on a field/subfield and > framework basis...that would be much easier to maintain! > > I don't know if that paragraph makes sense to anyone else...but I like this > idea. It's easier to maintain exceptions rather than a whole comprehensive > list, which can be dynamically generated. > > Bernardo, if you could post an up-to-date XML file with that MARC data I > mentioned above, I would be willing to take a look at creating a XSLT that > adds the relevant Koha data and produces a final "Koha MARC framework" XML > document in the end. > On the attached file there are XMLs for tags and subtags, with/without Koha proper fields. They are at least well formed (xmllint --noout file.xml) There's another thing to take into account: when MySQL 5.6 arrives to new debian/ubuntu versions, we'll have a problem. 5.6 client complains a lot to be used on CLI with password as argument, so Koha installer fails miserably because it uses 'mysql' command to load mandatory/optional values in database. Another reason to find other way. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #4 from Bernardo Gonzalez Kriegel --- Created attachment 29810 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29810&action=edit MARC21 bibliographic XML version Sample MARC21 as XML. There are a file for tags and another for subtags. Also there are complete and simple ones. Complete have all attributes, simple ones with only marc specific values. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #3 from David Cook --- Sorry for my delayed response! Competing priorities... (In reply to Bernardo Gonzalez Kriegel from comment #2) > If the field/subfield is missing, adding it as 'hidden' does not affect > present fw. And you can always preserve liblibrarian/libopac as they are... > except when they change, as in last update (Bug 12435, bib 347$f). > > As in other updates I proposed, I never try to update existing fw, for > the problems that you commented. > It's a tough one. I'm thinking that there should be some mechanism for updating an existing framework. It might need to be triggered by library staff, so that they understand what is being updated, but there should be something there. Otherwise, if you're on Koha for a long time, you'll get stale frameworks. Since Koha used outdated frameworks for a long time, it's possible (re: probable) that many Koha users are using very out of date frameworks that need to be updated in a user-friendly way. > And there is a bigger problem: translations > I haven't done anything with translations, but I'm in favour of anything that makes translations easier. > > We could actually just store this MARC-specific data in the YAML file, and > > then build the Default and other Custom frameworks based on a Perl script, > > which would probably be easier to maintain than the SQL that we maintain > > now. > > If it's agreed to store fw in other format, I propose we use XML (example > attached). And the reason is translations. > I'm happy to go with XML. I suppose my only reservations are about the brittleness of XML and processing time involved in parsing a huge XML file. I suppose the brittleness shouldn't be a factor...we could validate the XML before committing it and parsing errors are very loud so there probably is no danger there. As for processing time...it's not like installing or updating frameworks would be a frequent operation so lengthy processing time might not matter (if it is even a thing). > We need a way to ease the translation of all configuration and sample data > we have as SQL. I'm working intermittently on this, now I'm stuck with > sample notices, but have worked out most sql files as XML and a script to > build sql from them. And unless we propose another way, we need translated > sql files at web install time. It's not practical that each language other > than english have a lang dir commited to git. > Now that I think about it... are "liblibrarian" and "libopac" the only fields that need to be translated? > Talking again about frameworks, Bug 10963 is waiting for a review :) It looks good! I'll have to look at it soon. -- More thoughts that I'm having... marc_tag_structure: 1) MARC: tagfield, liblibrarian, libopac, repeatable 2) Koha: mandatory, authorised_value, frameworkcode marc_subfield_structure: 1) MARC: tagfield, tagsubfield, liblibrarian, libopac, repeatable, maxlength 2) Koha: mandatory, kohafield, tab, authorised_value, authtypecode, value_builder, isurl, hidden, frameworkcode, seealso, link, defaultvalue I'm wondering if it makes sense to maintain 2 separate sources of data. The first XML file would be the data that comes from the actual MARC specification (denoted by the "MARC:" tag above. The second XML file would be what Koha should do with the incoming data. -- Oooh. Brain wave...we could use XSLT! We'd have a XML file containing the MARC specification data. Then we pass that through an XSLT which adds the relevant "Koha" data. Think about it...in most cases those "Koha" fields are "null" or "". The most important fields are probably "tab" and "hidden". If we use defaults for most cases, but define exceptions in the XSLT on a field/subfield and framework basis...that would be much easier to maintain! I don't know if that paragraph makes sense to anyone else...but I like this idea. It's easier to maintain exceptions rather than a whole comprehensive list, which can be dynamically generated. Bernardo, if you could post an up-to-date XML file with that MARC data I mentioned above, I would be willing to take a look at creating a XSLT that adds the relevant Koha data and produces a final "Koha MARC framework" XML document in the end. >From there...we could use scripts to transform XML to SQL or to parse the XML and do the database updates via Perl. Anyway...just some thoughts for now. Have to run! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 Bernardo Gonzalez Kriegel changed: What|Removed |Added CC||bgkrie...@gmail.com --- Comment #2 from Bernardo Gonzalez Kriegel --- Created attachment 29316 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29316&action=edit Field 650 as XML (In reply to David Cook from comment #0) > We need better ways to manage MARC frameworks for authority and > bibliographic records (for new and existing instances). Agreed > > However, that puzzle can probably be solved in a few ways: > > 1) Inform users that standard fields/subfields will be periodically > added/updated in the frameworks; or > 2) Create a "Refresh" type button next to each framework which puts the > updating of frameworks into the hands of library staff If the field/subfield is missing, adding it as 'hidden' does not affect present fw. And you can always preserve liblibrarian/libopac as they are... except when they change, as in last update (Bug 12435, bib 347$f). As in other updates I proposed, I never try to update existing fw, for the problems that you commented. And there is a bigger problem: translations > We could actually just store this MARC-specific data in the YAML file, and > then build the Default and other Custom frameworks based on a Perl script, > which would probably be easier to maintain than the SQL that we maintain now. If it's agreed to store fw in other format, I propose we use XML (example attached). And the reason is translations. We need a way to ease the translation of all configuration and sample data we have as SQL. I'm working intermittently on this, now I'm stuck with sample notices, but have worked out most sql files as XML and a script to build sql from them. And unless we propose another way, we need translated sql files at web install time. It's not practical that each language other than english have a lang dir commited to git. I think that using YAML or XML is the same, in this case. Talking again about frameworks, Bug 10963 is waiting for a review :) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=1289 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11068 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=9678 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=3841 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=10881 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11654 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 --- Comment #1 from David Cook --- After reading bug 6007, I think perhaps it would be a good idea to have 'Display only used Tags/Subfields' on by default (provided that it's fixed/accurate). That could solve the potential problem of unused fields/subfields showing up when library staff are editing their existing frameworks (where they might have deleted a lot of standard fields/subfields). -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=6007 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11330 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=10963 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11030 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11021 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=11586 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=9826 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 12477] We need better ways to manage MARC Frameworks
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12477 David Cook changed: What|Removed |Added See Also||http://bugs.koha-community. ||org/bugzilla3/show_bug.cgi? ||id=10488 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
