Kyle Banerjee wrote:
On Mon, Oct 25, 2010 at 12:38 PM, Tim Spalding <t...@librarything.com> wrote:
Does processing speed of something matter anymore? You'd have to be
doing a LOT of processing to care, wouldn't you?
Data migrations and data dumps are a common use case. Needing to break or
make hundreds of thousands or millions of records is not uncommon.
kyle
To make this concrete, we processes the MARC records from 14 separate
ILS's throughout the University of Wisconsin System. We extract, sort on
OCLC number, dedup and merge pieces from any campus that has a record
for the work. The MARC that we then index and display here
http://forward.library.wisconsin.edu/catalog/ocm37443537?school_code=WU
is not identical to the version of the MARC record from any of the 4
schools that hold it.
We extract 13 million records and dedup down to 8 million every week.
Speed is paramount.
-sm
--
Stephen Meyer
Library Application Developer
UW-Madison Libraries
436 Memorial Library
728 State St.
Madison, WI 53706
sme...@library.wisc.edu
608-265-2844 (ph)
"Just don't let the human factor fail to be a factor at all."
- Andrew Bird, "Tables and Chairs"
