Or maybe a better way to ask this. Have people found a good way to export a large number of marc records within Evergreen? We found the staff client way to do it. And it processes files of 5-10,000 records at a time. But if we want to do 1 million records, let's say, it's a bit tedious. So then I found the marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=>. But that crashed our server doing it with 3,000 records at a time. We have ideas on how to modify the process, and it's not terrible using the staff client way, but I figure this must be a somewhat common task that others have good solutions for? Anyone willing to share 🙂?
Thanks, Brian Brian Holda Library Technology Manager Hekman Library Calvin University (616) 526-8673 [cid:e83c175b-fa22-41ee-b0e4-dc4102c98d50]<https://library.calvin.edu/> ________________________________ From: Evergreen-dev <[email protected]> on behalf of Brian Holda via Evergreen-dev <[email protected]> Sent: Thursday, October 5, 2023 4:25 PM To: Evergreen Development Discussion List <[email protected]> Subject: [Evergreen-dev] marc_export - apache crashes Hi all, Not sure if it's user error or something else going on, so wanted to see if any of you all have experience using marc_export script<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.evergreen-2Dils.org_3.2_-5Fmarc-5Fexport-5Fexporting-5Fbibliographic-5Frecords-5Finto-5Fmarc-5Ffiles.html&d=DwMGaQ&c=4rZ6NPIETe-LE5i2KBR4rw&r=rB3XDC6iCWGkjZtiGXbRHlEfGQP12yvXoVpChsQG6IY&m=PZmft0gaDkWQJ-PsQAmuzLbFyoxYjph24cGK4vqaTEXFnORZ9vEDMUnFRbuzb4np&s=28sCuznEp6eX3W0zi51ryW-NBwTZ0P8RjE4l6oEL_rE&e=> and had similar problems. In brief: * Tue, 5pm - I ran the following test (this is for a file of 3,100 records). This took about 30 sec. and successfully created the export file without any noticeable effects on our apache2 server: cat /home/opensrf/marc-test.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > exported_files.xml * Wed, 11:40am- I ran what I thought was essentially the same test (for the same file of 3,100 records). This also took about 30 sec. and successfully created the export file. However, 8 min. later apache crashed and had to be restarted. In the error log, it said "couldn't grab the accept mutex" immediately before crashing. Here's the code I ran: cat /tmp/marc-output/marc1.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /tmp/marc-output/exported-marc1.xml * Wed, 4pm- I ran essentially the same command (for the same file of 3,100 records), but without using the tmp​ folder. This time it stalled and after waiting a few minutes we pressed ctrl​ + c​ which I assumed stopped everything cleanly, as it returned me to the command prompt. However, at 4:50pm apache quit again, with the same "couldn't grab the accept mutex" messages beforehand. Here's the code I ran this time: cat /home/opensrf/marc2.txt | marc_export --reporter -i -c /openils/conf/opensrf_core.xml \ -x /openils/conf/fm_IDL.xml -f XML --timeout 5 > /home/opensrf/exported-marc2.xml Anyone know what might be happening here? Brian Holda Library Technology Manager Hekman Library Calvin University (616) 526-8673 [cid:4e579103-9d9b-445c-a644-73d36151b8e7]<https://library.calvin.edu/>
_______________________________________________ Evergreen-dev mailing list [email protected] http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev
