Re: [OPEN-ILS-DEV] 3.0.0-betat1 database upgrade and reingest

[Jason Stephenson]

Hi, again.

Just a follow up email to make it known that I have pushed a branch
named evergreen-3.0-compatibility to the evergreen_utilities on github.
This branch adds a --skip-display option to pingest.pl. This should make
it completely compatible with Evergreen 3.0's changes.

Cheers,
Jason

Re: [OPEN-ILS-DEV] 3.0.0-betat1 database upgrade and reingest

[Jason Stephenson]

Josh,

I have pushed some changes to the github repo for Evergreen utilities
that make pingest.pl use named parameters. This change does not include
in the skip_display option added in 3.0, so you may still have some
issues running it.

I have also not tested the modified script as a whole, yet. I did test
individual calls to the metabib.reingest_field_entries and
metabib.reingest_record_attribrutes functions in a test database.

I plan to make a branch for a version of pingest.pl that works better
with Evergreen 3.0. This version will add a skip display option to set
the skip_display parameter on metabib.reingest_field_entries. I will not
merge this branch into the master branch of the Evergreen utilities
until after we upgrade to 3.0 at C/W MARS. I will have a use for this
version though as I will set up a 3.0 test system in a week or so.

I will send another email to the list when I have made that branch, so
you'll know to look for it.

Cheers,
Jason

Re: [OPEN-ILS-DEV] 3.0.0-betat1 database upgrade and reingest

[Jason Stephenson]

On 09/21/2017 03:32 PM, Josh Stompro wrote:
> After the upgrade I started a reingest using the pingest.pl tool, since
> I skipped all the skippable reingests during the upgrade, and I received
> two instances of the following error.  I’m wondering if this is anything
> to worry about?  Or is this just a timing issue, maybe two threads of
> pingest.pl were both trying to create the same browse creator entry?  I
> used small batches of 250 records, so maybe that is it? 
> 
>  
> 
> DBD::Pg::st execute failed: ERROR:  duplicate key value violates unique
> constraint "browse_entry_sort_value_value_key"
> 
> DETAIL:  Key (sort_value, value)=(glasrud clarence a creator, Glasrud,
> Clarence A. creator) already exists.
> 
> CONTEXT:  SQL statement "INSERT INTO metabib.browse_entry
> 
>     ( value, sort_value ) VALUES
> 
>     ( value_prepped, ind_data.sort_value )"
> 
> PL/pgSQL function
> metabib.reingest_metabib_field_entries(bigint,boolean,boolean,boolean,boolean,integer[])
> line 86 at SQL statement at ./pingest.pl line 272.
> 
> metabib.reingest_metabib_field_entries failed for record 8544 at
> ./pingest.pl line 275.

I'm curious to know what options you used when you ran pingest.pl and if
it is actually a "stock" version of pingest.pl. That above error should
not happen with the call on line 272. It deliberately skips doing the
browse ingest.

That said, I have not run pingest on a 3.0 database so something may
have changed in the reingest_metabib_field_entries function that leads
to this error.

> I re-ran the pingest just for the records mentioned and that did take
> care of adding the correct entries to metabib.browse_entry_def_map.
> 
>  
> 
> Does each batch in pingest.pl run in its own transaction?  Would this
> error mean that all the changes for that batch of 250 records were
> rolled back?

pingest.pl doesn't use transactions, so only the failed records need to
be run again.

pingest.pl splits the input into batches for the search, facets, and
record attribute ingest which can be done in parallel. The browse ingest
is done in one giant batch of all the records because of errors similar
to the one you cite above when two batches try to update a record on the
same authority.

Like I said, I'm curious what options you used and what has changed in
Evergreen 3.0.

> 
>  
> 
> Thanks
> 
>  
> 
> Lake Agassiz Regional Library - Moorhead MN larl.org
> 
> Josh Stompro | Office 218.233.3757 EXT-139
> 
> LARL IT Director | Cell 218.790.2110 
> 
>  
>