Josh, Unfortunately, there are parts of the ingest process that cannot be done in parallel, and pingest only does a subset of a full ingest anyway.
What I am now working on will update the biblio.record_entry in such a way that a full ingest will be done, including call the maintain_901 function in the database (rather than duplicating it in the script). This involves messing with internal flags and a plethora of command line options to skip various portions of the ingest. This version, which I am calling ngingest at the moment, cannot be parallelized without a great deal more smarts. I will share it to my Evergreen utilities repository on github when I'm convinced it is working. Cheers, Jason On 04/20/2017 12:34 PM, Josh Stompro wrote: > Jason, I would be interested in seeing your script. I wonder if your > pingest.pl script that many people use for reingest could be extended with an > option to trigger the maintain_901 function as another method? Some sort of > "trigger bib update" option? That could handle spreading out the load/locks > also. Maybe that wouldn't allow the user to see the encoding errors though? > > Maybe the encoding errors should be tested for a different way though, > instead of trying to combine that with this update? > > Josh Stompro - LARL IT Director > > > -----Original Message----- > From: Open-ils-general > [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of > Jason Stephenson > Sent: Thursday, April 20, 2017 7:20 AM > To: Evergreen Discussion Group > Subject: Re: [OPEN-ILS-GENERAL] Upgrading from 2.10 to 2.11 error questions > > Hi, Jesse. > > I won't repeat what Dan said, but I'll add that with a change to an internal > flag or two this update will also take care of any ingests required. I think > there's a browse or facet ingest required for 2.11. > > I'm preparing to upgrade from 2.10 to 2.12, and I've run into a similar issue > as you with the maintain_901 function and some records with "bad" > characters. I'm working on a Perl script to pull the MARC out of the database > and do the maintain_901 work in a function in the script. The advantage of > this is two fold: > > #1 I can trap any errors that occur and output a warning message with the > record id that causes the error so it can later be fixed. > > #2 By modifying the MARC and then updating the record in the database, I can > get the ingests to trigger without altering any database flags. > > It won't be fast, but it won't have the downside of potentially locking > significant portions of the biblio.record_entry table. Updating 1 row at a > time should keep it at row locks, or page locks at worst. > > If there is enough interest expressed on the list, I could share this code on > github or elsewhere. > > Cheers, > Jason > > On 04/19/2017 04:22 PM, Jesse McCarty wrote: >> Hello Everyone, >> >> >> >> We are preparing for our spring upgrade to Evergreen, moving from >> 2.10.3 to 2.11.3 and ran into one little bump. As part of the DB >> upgrade, there is an update setting the 901$s for bib records. First >> question, as seen in the attached screen shot, this threw a bunch of >> 'no mapping found for....' Errors. Can this be safely ignored and >> proceed with running the system after upgrading with no issues (we >> haven't seen any issue in our testing)? >> >> >> >> The second, this update seem to take longer than 24 hours. With that >> in mind would we be able to process the entire upgrade, then use >> Evergreen in daily production while this DB update finishes in the >> background? Or does this need to be 100% complete before allowing >> library's connection to the system? >> >> >> >> Thanks in advance, >> >> >> >> Jesse McCarty >> >> City of Burlington >> >> IT Technical Assistant >> >> >>