When exposing sets of MARC records as linked data, do you think it is better to
expose them in batch (collection) files or as individual RDF serializations? To
bastardize the Bard — “To batch or not to batch? That is the question.”
Suppose I am a medium-sized academic research library. Suppose my collection is
comprised of approximately 3.5 million bibliographic records. Suppose I want to
expose those records via linked data. Suppose further that this will be done by
“simply” making RDF serialization files (XML, Turtle, etc.) accessible via an
HTTP filesystem. No scripts. No programs. No triple stores. Just files on an
HTTP file system coupled with content negotiation. Given these assumptions,
would you:
1. create batches of MARC records, convert them to MARCXML
and then to RDF, and save these files to disc, or
2. parse the batches of MARC record sets into individual
records, convert them into MARCXML and then RDF, and
save these files to disc
Option #1 would require heavy lifting against large files, but the number of
resulting files to save to disc would be relatively few — reasonably managed in
a single directory on disc. On the other hand, individual URIs pointing to
individual serializations would not be accessible. They would only be
accessible by retrieving the collection file in which they reside. Moreover, a
mapping of individual URIs to collection files would need to be maintained.
Option #2 would be easier on the computing resources because processing little
files is generally easier than processing bigger ones. On the other hand, the
number of files generated by this option is not easily be managed without the
use of a sophisticated directory structure. (It is not feasible to put 3.5
million files in a single directory.) But I would still need to create a
mapping from URI to directory.
In either case, I would probably create a bunch of site map files denoting the
locations of my serializations — YAP (Yet Another Mapping).
I’m leaning towards Option #2 because individual URIs could be resolved more
easily with “simple” content negotiation.
(Given my particular use case — archival MARC records — I don’t think I’d
really have more than a few thousand items, but I’m asking the question on a
large scale anyway.)
—
Eric Morgan
