Re: [Archivesspace_Users_Group] PUI inheritance configuration

[James Bullen]

Hi Cory,

It’s been some years since this was in my head, but it looks good.

The :inherit_if on the first rule should be selecting all subjects that have at 
least one term with a ‘genreform’ type.


Cheers,
James


> On Feb 20, 2020, at 1:37 AM, Cory Nimer  wrote:
> 
> James,
>  
> To my knowledge, a complete reindex was done following the changes (other 
> changes in note inheritance did appear as anticipated). However, the subject 
> inheritance did not seem to work. Are the inheritance statements below 
> structured correctly, or are there other reasons that these elements would 
> not be visible in the interface?
>  
> Best,
>  
> Cory
>  
> From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
>  
>  > On Behalf 
> Of James Bullen
> Sent: Tuesday, February 18, 2020 6:49 PM
> To: Archivesspace Users Group 
>  >
> Subject: Re: [Archivesspace_Users_Group] PUI inheritance configuration
>  
>  
> Hi Cory,
>  
> Have you reindexed?
>  
> The PUI runs exclusively from the PUI index so these changes won’t take 
> effect until the relevant records have been reindexed.
>  
>  
> Cheers,
> James
>  
> 
> 
> On Feb 19, 2020, at 2:26 AM, Cory Nimer  > wrote:
>  
> We are continuing to tweak our PUI configuration, and are interested in 
> testing inclusion of subject terms. Based on the configuration template, our 
> IT staff have attempted to add the code below but it does not appear to be 
> working after a restart and reindex: 
>  
> {
>   :property => 'subjects',
>   :inherit_if => proc {|json| json.select {|j| ! 
> j['_resolved']['terms'].select { |t| t['term_type'] == 'genreform'}.empty? } 
> },
>   :inherit_directly => true
> }, 
> {
>   :property => 'subjects',
>   :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
>   :inherit_if => proc {|json| json.select {|j| ! 
> j['_resolved']['terms'].select { |t| t['term_type'] == 'topical'}.empty? } },
>   :inherit_directly => true
> }, 
> {
>   :property => 'subjects',
>   :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
>   :inherit_if => proc {|json| json.select {|j| ! 
> j['_resolved']['terms'].select { |t| t['term_type'] == 'title'}.empty? } },
>   :inherit_directly => true
> }, 
> {
>   :property => 'subjects',
>   :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
>   :inherit_if => proc {|json| json.select {|j| ! 
> j['_resolved']['terms'].select { |t| t['term_type'] == 'geogname'}.empty? } },
>   :inherit_directly => true
> }, 
>  
> Has anyone else successfully included subject terms in their PUI inheritance 
> configuration, or have suggestions for what might be wrong? We are currently 
> running version 2.5.1.
>  
> Thanks,
>  
> Cory Nimer
> University Archivist
> Brigham Young University
> 801-422-6091
>  
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org 
> 
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group 
> 
> 
> 
> !DSPAM:5e4c053f111932459259872!
>  
> !DSPAM:5e4d482a179227770718730! 
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org 
> 
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group 
> 
> 
> 
> !DSPAM:5e4d482a179227770718730!

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

[Joshua D. Shaw]

One additional little bit of info that may complicate things. It seems like the 
pause will not shut down an indexer run midway. It will only take effect after 
the current run is complete. At least that seems to be the case. For the import 
jobs, that may make things a bit dicey if a large index run is in progress when 
the import job kicks off, since you could still get that sync happening. I 
guess you could wait for the indexer to complete (not sure how to get that 
status) and then kick off the import job? Or, perhaps the import jobs should 
add a final step that iterates through all of the created objects and sets the 
mtime to the import job completion time?

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Joshua 
D. Shaw 
Sent: Wednesday, February 19, 2020 9:40 AM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

I think I've got it going by adding a little loop that checks the indexer state 
via the response body of the indexer (ie does it contain the string "paused") 
and then sending a "put" with the duration parameter set to either pause (you 
can specify an exact duration or just let it default to 900 seconds) or resume 
(duration = 0). Things to note:

1) For some reason, the indexer actually listens at 
AppConfig[:indexer_url]/aspace-indexer/ not just AppConfig[:indexer_url]
2) There's no nice ASHTTP wrapper for put, so you have to construct the 
Net::HTTP for the put yourself

I don't see why similar logic couldn't be incorporated into any import job so 
that the import has a chance to finish up before the indexer runs again, 
preventing the sync issues in ANW-902.

I've got about 280k objects to check and update, so I'll see if I run into any 
indexer issues once the job is completed. The only thing I've seen that may be 
related to that is a snapshot failure when doing a large index run (full or 
otherwise), but I don't think I've ever seen it completely fail due to a commit 
timeout. That almost sounds more like disk access or network (if you're running 
a separate SOLR instance).

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Andrew 
Morrison 
Sent: Wednesday, February 19, 2020 8:32 AM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

I'd be interested in hearing if you get this to work, because it could be 
useful in fixing this issue:

https://archivesspace.atlassian.net/browse/ANW-902

Also, if you're making a truly mammoth update, which will be followed by a 
re-index of nearly everything, you might want to consider increasing the 
AppConfig[:indexer_solr_timeout_seconds] config setting. It may be our 
infrastructure, but I've found that Solr commit's phase can take so long that 
ArchivesSpace times out before it finishes, causing it to start the whole 
re-index again from scratch. We've set it to 1800 to avoid this, but YMMV.

Andrew.


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Joshua 
D. Shaw 
Sent: 19 February 2020 13:05
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

Thanks, James. I glanced at that, but somehow didn't realize those were 
endpoints I could hit. I'll give it a go!

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of James 
Bullen 
Sent: Tuesday, February 18, 2020 7:16 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?


Hi Joshua,

I haven’t used it, but I see these endpoints in indexer/app/main.rb

  get "/" do
if IndexerCommon.paused?
  "Indexers paused until 
#{IndexerCommon.class_variable_get(:@@paused_until)}"
else
  "Running every #{AppConfig[:solr_indexing_frequency_seconds].to_i} 
seconds. "
end
  end

  # this pauses the indexer so that bulk update and migrations can happen
  # without bogging down the server
  put "/" do
duration = params[:duration].nil? ? 900 : params[:duration].to_i
IndexerCommon.pause duration
"#{IndexerCommon.class_variable_get(:@@paused_until)}"
  end


Seems to do what you want.


Cheers,
James


On Feb 19, 2020, at 6:29 AM, Joshua D. Shaw 
mailto:joshua.d.s...@dartmouth.edu>> wrote:

Hey all-

I writing a job that may take a *long* time (hours) to complete which will be 
updating a *lot* of AO records. I'm wondering if there's a way to pause the 
Indexer during a job so that I 

Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

[Joshua D. Shaw]

I think I've got it going by adding a little loop that checks the indexer state 
via the response body of the indexer (ie does it contain the string "paused") 
and then sending a "put" with the duration parameter set to either pause (you 
can specify an exact duration or just let it default to 900 seconds) or resume 
(duration = 0). Things to note:

1) For some reason, the indexer actually listens at 
AppConfig[:indexer_url]/aspace-indexer/ not just AppConfig[:indexer_url]
2) There's no nice ASHTTP wrapper for put, so you have to construct the 
Net::HTTP for the put yourself

I don't see why similar logic couldn't be incorporated into any import job so 
that the import has a chance to finish up before the indexer runs again, 
preventing the sync issues in ANW-902.

I've got about 280k objects to check and update, so I'll see if I run into any 
indexer issues once the job is completed. The only thing I've seen that may be 
related to that is a snapshot failure when doing a large index run (full or 
otherwise), but I don't think I've ever seen it completely fail due to a commit 
timeout. That almost sounds more like disk access or network (if you're running 
a separate SOLR instance).

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Andrew 
Morrison 
Sent: Wednesday, February 19, 2020 8:32 AM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

I'd be interested in hearing if you get this to work, because it could be 
useful in fixing this issue:

https://archivesspace.atlassian.net/browse/ANW-902

Also, if you're making a truly mammoth update, which will be followed by a 
re-index of nearly everything, you might want to consider increasing the 
AppConfig[:indexer_solr_timeout_seconds] config setting. It may be our 
infrastructure, but I've found that Solr commit's phase can take so long that 
ArchivesSpace times out before it finishes, causing it to start the whole 
re-index again from scratch. We've set it to 1800 to avoid this, but YMMV.

Andrew.


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Joshua 
D. Shaw 
Sent: 19 February 2020 13:05
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

Thanks, James. I glanced at that, but somehow didn't realize those were 
endpoints I could hit. I'll give it a go!

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of James 
Bullen 
Sent: Tuesday, February 18, 2020 7:16 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?


Hi Joshua,

I haven’t used it, but I see these endpoints in indexer/app/main.rb

  get "/" do
if IndexerCommon.paused?
  "Indexers paused until 
#{IndexerCommon.class_variable_get(:@@paused_until)}"
else
  "Running every #{AppConfig[:solr_indexing_frequency_seconds].to_i} 
seconds. "
end
  end

  # this pauses the indexer so that bulk update and migrations can happen
  # without bogging down the server
  put "/" do
duration = params[:duration].nil? ? 900 : params[:duration].to_i
IndexerCommon.pause duration
"#{IndexerCommon.class_variable_get(:@@paused_until)}"
  end


Seems to do what you want.


Cheers,
James


On Feb 19, 2020, at 6:29 AM, Joshua D. Shaw 
mailto:joshua.d.s...@dartmouth.edu>> wrote:

Hey all-

I writing a job that may take a *long* time (hours) to complete which will be 
updating a *lot* of AO records. I'm wondering if there's a way to pause the 
Indexer during a job so that I can let the Indexer do its thing*after* the job 
completes. I know I can toggle the AppConfig value for the indexer and do a 
stop/start for the app, but ideally I'd like to do the pause/resume of the 
Indexer while the job runs.

I could also set this up as a migration, but the updates include a bunch of 
tables (I'm adding an instance to AOs which meet certain criteria) and I'd 
prefer to use the API to do things to be safe.

Any thoughts on pausing the Indexer during a job, or do I bite the bullet and 
do this as a migration?

Thanks!
Joshua

___
Joshua Shaw (he, him)
Technology Coordinator
Rauner Special Collections Library & Digital Library Technologies Group
Dartmouth College
603.646.0405
!DSPAM:5e4c3b1e193891489818497! ___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org

Re: [Archivesspace_Users_Group] PUI inheritance configuration

[Cory Nimer]

James,

To my knowledge, a complete reindex was done following the changes (other 
changes in note inheritance did appear as anticipated). However, the subject 
inheritance did not seem to work. Are the inheritance statements below 
structured correctly, or are there other reasons that these elements would not 
be visible in the interface?

Best,

Cory

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of James 
Bullen
Sent: Tuesday, February 18, 2020 6:49 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] PUI inheritance configuration


Hi Cory,

Have you reindexed?

The PUI runs exclusively from the PUI index so these changes won’t take effect 
until the relevant records have been reindexed.


Cheers,
James



On Feb 19, 2020, at 2:26 AM, Cory Nimer 
mailto:cory_ni...@byu.edu>> wrote:

We are continuing to tweak our PUI configuration, and are interested in testing 
inclusion of subject terms. Based on the configuration template, our IT staff 
have attempted to add the code below but it does not appear to be working after 
a restart and reindex:

{
  :property => 'subjects',
  :inherit_if => proc {|json| json.select {|j| ! j['_resolved']['terms'].select 
{ |t| t['term_type'] == 'genreform'}.empty? } },
  :inherit_directly => true
},
{
  :property => 'subjects',
  :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
  :inherit_if => proc {|json| json.select {|j| ! j['_resolved']['terms'].select 
{ |t| t['term_type'] == 'topical'}.empty? } },
  :inherit_directly => true
},
{
  :property => 'subjects',
  :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
  :inherit_if => proc {|json| json.select {|j| ! j['_resolved']['terms'].select 
{ |t| t['term_type'] == 'title'}.empty? } },
  :inherit_directly => true
},
{
  :property => 'subjects',
  :skip_if => proc {|json| ['file', 'item'].include?(json['level']) },
  :inherit_if => proc {|json| json.select {|j| ! j['_resolved']['terms'].select 
{ |t| t['term_type'] == 'geogname'}.empty? } },
  :inherit_directly => true
},

Has anyone else successfully included subject terms in their PUI inheritance 
configuration, or have suggestions for what might be wrong? We are currently 
running version 2.5.1.

Thanks,

Cory Nimer
University Archivist
Brigham Young University
801-422-6091

!DSPAM:5e4c053f111932459259872! ___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group


!DSPAM:5e4c053f111932459259872!

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

[Andrew Morrison]

I'd be interested in hearing if you get this to work, because it could be 
useful in fixing this issue:

https://archivesspace.atlassian.net/browse/ANW-902

Also, if you're making a truly mammoth update, which will be followed by a 
re-index of nearly everything, you might want to consider increasing the 
AppConfig[:indexer_solr_timeout_seconds] config setting. It may be our 
infrastructure, but I've found that Solr commit's phase can take so long that 
ArchivesSpace times out before it finishes, causing it to start the whole 
re-index again from scratch. We've set it to 1800 to avoid this, but YMMV.

Andrew.


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Joshua 
D. Shaw 
Sent: 19 February 2020 13:05
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

Thanks, James. I glanced at that, but somehow didn't realize those were 
endpoints I could hit. I'll give it a go!

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of James 
Bullen 
Sent: Tuesday, February 18, 2020 7:16 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?


Hi Joshua,

I haven’t used it, but I see these endpoints in indexer/app/main.rb

  get "/" do
if IndexerCommon.paused?
  "Indexers paused until 
#{IndexerCommon.class_variable_get(:@@paused_until)}"
else
  "Running every #{AppConfig[:solr_indexing_frequency_seconds].to_i} 
seconds. "
end
  end

  # this pauses the indexer so that bulk update and migrations can happen
  # without bogging down the server
  put "/" do
duration = params[:duration].nil? ? 900 : params[:duration].to_i
IndexerCommon.pause duration
"#{IndexerCommon.class_variable_get(:@@paused_until)}"
  end


Seems to do what you want.


Cheers,
James


On Feb 19, 2020, at 6:29 AM, Joshua D. Shaw 
mailto:joshua.d.s...@dartmouth.edu>> wrote:

Hey all-

I writing a job that may take a *long* time (hours) to complete which will be 
updating a *lot* of AO records. I'm wondering if there's a way to pause the 
Indexer during a job so that I can let the Indexer do its thing*after* the job 
completes. I know I can toggle the AppConfig value for the indexer and do a 
stop/start for the app, but ideally I'd like to do the pause/resume of the 
Indexer while the job runs.

I could also set this up as a migration, but the updates include a bunch of 
tables (I'm adding an instance to AOs which meet certain criteria) and I'd 
prefer to use the API to do things to be safe.

Any thoughts on pausing the Indexer during a job, or do I bite the bullet and 
do this as a migration?

Thanks!
Joshua

___
Joshua Shaw (he, him)
Technology Coordinator
Rauner Special Collections Library & Digital Library Technologies Group
Dartmouth College
603.646.0405
!DSPAM:5e4c3b1e193891489818497! ___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group


!DSPAM:5e4c3b1e193891489818497!

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?

[Joshua D. Shaw]

Thanks, James. I glanced at that, but somehow didn't realize those were 
endpoints I could hit. I'll give it a go!

Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of James 
Bullen 
Sent: Tuesday, February 18, 2020 7:16 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Method to Pause Indexer during Job Run?


Hi Joshua,

I haven’t used it, but I see these endpoints in indexer/app/main.rb

  get "/" do
if IndexerCommon.paused?
  "Indexers paused until 
#{IndexerCommon.class_variable_get(:@@paused_until)}"
else
  "Running every #{AppConfig[:solr_indexing_frequency_seconds].to_i} 
seconds. "
end
  end

  # this pauses the indexer so that bulk update and migrations can happen
  # without bogging down the server
  put "/" do
duration = params[:duration].nil? ? 900 : params[:duration].to_i
IndexerCommon.pause duration
"#{IndexerCommon.class_variable_get(:@@paused_until)}"
  end


Seems to do what you want.


Cheers,
James


On Feb 19, 2020, at 6:29 AM, Joshua D. Shaw 
mailto:joshua.d.s...@dartmouth.edu>> wrote:

Hey all-

I writing a job that may take a *long* time (hours) to complete which will be 
updating a *lot* of AO records. I'm wondering if there's a way to pause the 
Indexer during a job so that I can let the Indexer do its thing*after* the job 
completes. I know I can toggle the AppConfig value for the indexer and do a 
stop/start for the app, but ideally I'd like to do the pause/resume of the 
Indexer while the job runs.

I could also set this up as a migration, but the updates include a bunch of 
tables (I'm adding an instance to AOs which meet certain criteria) and I'd 
prefer to use the API to do things to be safe.

Any thoughts on pausing the Indexer during a job, or do I bite the bullet and 
do this as a migration?

Thanks!
Joshua

___
Joshua Shaw (he, him)
Technology Coordinator
Rauner Special Collections Library & Digital Library Technologies Group
Dartmouth College
603.646.0405
!DSPAM:5e4c3b1e193891489818497! ___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group


!DSPAM:5e4c3b1e193891489818497!

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group