Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089 error

[Tom Hanstra]

Yes, that is just Solr. But if this is a new server could you just turn off
Splunk for a while just to see that it is not the conflict?  Nothing in
what you've shown us so far would suggest that there is a conflict on port
8089. But if Splunk might be the issue, try turning it off and then
starting ArchivesSpace

On Fri, May 19, 2023 at 11:22 AM Gadsby, Eric T.  wrote:

> ps -ef  | grep java =
>
>
>
> “java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem
> -XX:+ParallelRefProcEnabled
> -XX
> :MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch
> -XX:+ExplicitGCInvokesConcurrent
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filec
> ount=9,filesize=20M -Dsolr.jetty.inetaccess.includes=
> -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/var/solr/logs
> -Djetty.port=8983
> -DSTOP.PORT=7983
> -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow
> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs
> -Djetty.home
>  =/opt/solr/server
> -Dsolr.solr.home=/var/solr/data -Dsolr.data.home=
> -Dsolr.install.dir=/opt/solr
> -Dsolr.default.confdir=/opt/solr/server/solr/configsets/
> _default/conf -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k
> -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip
>
> egadsby+65612546  0 11:18 pts/000:00:00 grep --color=auto java”
>
>
>
> Right now it just seems to be Solr running…. Based on my read of this
> output.   I might be wrong.
>
>
>
> It is a new 3.3.1 server with a fresh install with the data base from an
> old server being imported from a SQL dump and then migrated using the
> script.  Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>
>
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Friday, May 19, 2023 at 11:14 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] Trying to spin up a new server
> and it won't load... Unhandled Java exception: java.io.IOException: Failed
> to bind to 0.0.0.0/0.0.0.0:8089 error
>
> [ *CAUTION*: This email is from outside of TU. Use caution before
> clicking links or opening attachments. If suspicious, report to
> phish...@towson.edu. ]
>
> When connected to the server, use the command:
>
> ps -ef  | grep java
>
> and look to see if there are java processes running. If there is an old
> one, you will have to stop/kill it before starting the new one.
>
> Is this a completely new server only running 3.3.1 ? If so, the solr
> process will also show up using that command.
>
>
>
> Tom
>
>
>
> On Fri, May 19, 2023 at 10:55 AM Gadsby, Eric T. 
> wrote:
>
> I wondered that, how would I check? Going to a browser yields nothing…
> that said my firewall rules may not be in place? Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>

Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089 error

[Tom Hanstra]

When connected to the server, use the command:

ps -ef  | grep java

and look to see if there are java processes running. If there is an old
one, you will have to stop/kill it before starting the new one.

Is this a completely new server only running 3.3.1 ? If so, the solr
process will also show up using that command.

Tom

On Fri, May 19, 2023 at 10:55 AM Gadsby, Eric T.  wrote:

> I wondered that, how would I check? Going to a browser yields nothing…
> that said my firewall rules may not be in place? Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>
>
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Friday, May 19, 2023 at 10:53 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] Trying to spin up a new server
> and it won't load... Unhandled Java exception: java.io.IOException: Failed
> to bind to 0.0.0.0/0.0.0.0:8089 error
>
> [ *CAUTION*: This email is from outside of TU. Use caution before
> clicking links or opening attachments. If suspicious, report to
> phish...@towson.edu. ]
>
> First thought would be that the new service is trying to start while the
> old one is still running. Could that be?
>
>
>
> Tom
>
>
>
> On Fri, May 19, 2023 at 10:44 AM Gadsby, Eric T. 
> wrote:
>
> Dear Friends,
>
>
>
> I am in the process of moving our old v2.7.1 to V3.3.1. I have been able
> to set up Solr, the application, move the data and start ArchivesSpace
> twice in our testing environment (CentOS8 Stream). Today I ran the same
> procedure in our production environment (RHEL8.5) and ArchivesSpace won't
> start.
>
>
>
> The log reports this error: "Unhandled Java exception:
> java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089" -- of at
> least I think that's the problem (I'm going to provide the whole log:
>
>
>
> "$ cat archivesspace.out
>
> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated
> in version 9.0 and will likely be removed in a future release.
>
> Loading ArchivesSpace configuration file from path:
> /opt/archivesspace/config/config.rb
>
> Loading ArchivesSpace configuration file from path:
> /opt/archivesspace/config/config.rb
>
> Loading ArchivesSpace configuration file from path:
> /opt/archivesspace/config/config.rb
>
> May 19, 2023 10:01:02 AM org.eclipse.jetty.util.log.Log initialized
>
> INFO: Logging initialized @7281ms to org.eclipse.jetty.util.log.Slf4jLog
>
> May 19, 2023 10:01:03 AM org.eclipse.jetty.server.Server doStart
>
> INFO: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git:
> 8da83308eeca865e495e53ef315a249d63ba9332; jvm 11.0.19+7-LTS
>
> May 19, 2023 10:01:03 AM
> org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
>
> INFO: DefaultSessionIdManager workerName=node0
>
> May 19, 2023 10:01:03 AM
> org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
>
> INFO: No SessionScavenger set, using defaults
>
> May 19, 2023 10:01:03 AM org.eclipse.jetty.server.session.HouseKeeper
> startScavenging
>
> INFO: node0 Scavenging every 60ms
>
> May 19, 2023 10:01:03 AM
> org.eclipse.jetty.server.handler.ContextHandler$Contextlog
>
> INFO: INFO: jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit
> Server VM 11.0.19+7-LTS on 11.0.19+7-LTS +jit [linux-x86_64]
>
> May 19, 2023 10:01:03 AM
> org.eclipse.jetty.server.handler.ContextHandler$Contextlog
>
> INFO: INFO: using a shared (threadsafe!) runtime
>
> uri:classloader:/jruby/rack/response.rb:294: warning: constant ::Fixnum is
> deprecated
>
> uri:classloader:/jruby/rack/core_ext.rb:26: warning: constant
> ::NativeExceptionis deprecated
>
> Loading Ar

Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089 error

[Tom Hanstra]

gt;invokeOther132:start at launcher/launcher.rb:90
>
>  RUBY$method$start_server$1 at launcher/launcher.rb:90
>
>call at
> org/jruby/internal/runtime/methods/CompiledIRMethod.java:80
>
>call at
> org/jruby/internal/runtime/methods/CompiledIRMethod.java:179
>
>cacheAndCall at
> org/jruby/runtime/callsite/CachingCallSite.java:397
>
>call at
> org/jruby/runtime/callsite/CachingCallSite.java:206
>
> invokeOther193:start_server at launcher/launcher.rb:144
>
>  RUBY$method$main$6 at launcher/launcher.rb:144
>
>call at
> org/jruby/internal/runtime/methods/CompiledIRMethod.java:156
>
>cacheAndCall at
> org/jruby/runtime/callsite/CachingCallSite.java:355
>
>call at
> org/jruby/runtime/callsite/CachingCallSite.java:144
>
> invokeOther303:main at launcher/launcher.rb:242
>
> RUBY$script at launcher/launcher.rb:242
>
> invokeWithArguments at
> java/lang/invoke/MethodHandle.java:710
>
>load at org/jruby/ir/Compiler.java:89
>
>   runScript at org/jruby/Ruby.java:1205
>
> runNormally at org/jruby/Ruby.java:1128
>
> runNormally at org/jruby/Ruby.java:1146
>
> runFromMain at org/jruby/Ruby.java:958
>
>   doRunFromMain at org/jruby/Main.java:400
>
> internalRun at org/jruby/Main.java:292
>
> run at org/jruby/Main.java:234
>
>main at org/jruby/Main.java:206"
>
>
>
> I have Solr and ArchivesSpace set to start as init scripts. I would
> appreciate any insight that folks could offer would be great. Thanks!
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>
>
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] ID display config options

[Tom Hanstra]

Does anyone have additional information or documentation on how to
implement this plugin? There is not much in the README file in the
repository.

Thanks,
Tom

On Tue, May 16, 2023 at 8:49 AM Erhan Tuskan  wrote:

> Yes, what Brian thinks must be true, at IISH we use the Cambridge plugin
> too and the Component Unique Identifiers are displayed in the PUI and staff
> client as well.
>
>
>
> Erhan,
>
>
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Brian
> Hoffman
> *Sent:* dinsdag 16 mei 2023 2:27
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] ID display config options
>
>
>
> Hi Jennifer,
>
>
>
> I believe that functionality was originally developed as a plugin –
> perhaps that explains why Cambridge has it working already?
>
>
>
> Brian
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Jennifer Brcka 
> *Date: *Monday, May 15, 2023 at 4:49 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Hi Brian,
>
> For clarification -- I'd seen IDs in Cambridge's public display and had
> asked Tom to toggle our instance on after that model. Is the tree display
> here <https://archivesearch.lib.cam.ac.uk/repositories/2/resources/8865>
> output by the PUI configuration option we're discussing? Or does something
> else make that ID display possible?
>
> The staff-side view presented in the pull request you referenced also
> looks like exactly what we're after. The work being done, any chance this
> option could be included in the 3.4 release? Thanks again,
>
> Jennifer
>
>
>
> On Mon, May 15, 2023 at 1:41 PM Brian Hoffman 
> wrote:
>
> Hi Tom,
>
>
>
> I don’t believe it has worked in 3.3.1 or any previous release. Yes, it
> should be working in 3.5 as the work has already been done:
>
>
>
> https://github.com/archivesspace/archivesspace/pull/2987
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Monday, May 15, 2023 at 1:15 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Brian:
>
>
>
> Is this a future version such as the released version of 3.4 or a future
> version like in 3.5 or later?
>
> I'm unclear how functionality in 3.3.1 would get "lost" in the move to
> 3.4.
>
>
>
> Thanks,
>
> Tom
>
>
>
> On Mon, May 15, 2023 at 1:00 PM Brian Hoffman 
> wrote:
>
> Hi Jennifer,
>
>
>
> There is an issue with the way those features were implemented, and they
> won’t work without some customization in version 3.4.0. There is a fix
> completed that will appear in a future version.
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Andrew Morrison 
> *Date: *Monday, May 15, 2023 at 12:27 PM
> *To: *archivesspace_users_group@lyralists.lyrasis.org <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Those two features are partially implemented in JavaScript. It is possible
> that your web browser has cached the old versions of the JavaScript files.
> Try doing a hard refresh
> <https://www.howtogeek.com/672607/how-to-hard-refresh-your-web-browser-to-bypass-your-cache/>
> or emptying your browser's cache.
>
> Andrew.
>
>
>
> On 15/05/2023 16:53, Jennifer Brcka wrote:
>
> Hi All,
>
>
>
> We've been testing 3.4.0-RC1 and haven't been able to get two
> configuration options to work:
>
>
> AppConfig[:display_identifiers_in_largetree_container] = true
>
> AppConfig[:pui_display_identifiers_in_resource_tree] = true
>
>
>
> Our database was reindexed after switching these to 'true.'  Any thoughts
> on what could be going on? Is there anything else we need to change to get
> this to work?
>
>
>
> Thanks,
>
> Jennifer
>
>
>
> --
>
> *Jennifer Brcka*
>
> *Archives Specialist*
> Archival Collections & Management, Hesburgh Li

Re: [Archivesspace_Users_Group] ID display config options

[Tom Hanstra]

Thanks, Brian.  I pretty much figured that that might be the case. Just a
hair too late.

Thanks, Erhan for pointing me to the plugin. We will pursue that for the
time being.

Tom

On Tue, May 16, 2023 at 9:19 AM Brian Hoffman 
wrote:

> Hi Tom,
>
>
>
> In this case, the work just happened to get done right after the cut-off
> point for the new release. The associated JIRA issue is still in “Ready for
> Testing” state, so it needs to be tested and accepted before it is
> considered “done”.
>
>
>
> https://archivesspace.atlassian.net/browse/ANW-1196
>
>
>
> Brian
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Tuesday, May 16, 2023 at 9:02 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Thanks for the clarification and sorry for the confusion.
>
>
>
> Again, I find myself not familiar enough with the process for
> ArchivesSpace releases and development. If the work is done (as shown in
> the pull request sent), what is barring this from being added into the
> upcoming release of 13.4? Seems like completed work could easily be added
> to the next release, but I'm probably being too simplistic?
>
>
>
> If it won't be added by 3.4, I'll look into the plugin. If someone has a
> URL for it, that would be great. Or I'll try Google.
>
>
>
> Thanks,
>
> Tom
>
>
>
> On Tue, May 16, 2023 at 8:49 AM Erhan Tuskan  wrote:
>
> Yes, what Brian thinks must be true, at IISH we use the Cambridge plugin
> too and the Component Unique Identifiers are displayed in the PUI and staff
> client as well.
>
>
>
> Erhan,
>
>
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Brian
> Hoffman
> *Sent:* dinsdag 16 mei 2023 2:27
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] ID display config options
>
>
>
> Hi Jennifer,
>
>
>
> I believe that functionality was originally developed as a plugin –
> perhaps that explains why Cambridge has it working already?
>
>
>
> Brian
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Jennifer Brcka 
> *Date: *Monday, May 15, 2023 at 4:49 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Hi Brian,
>
> For clarification -- I'd seen IDs in Cambridge's public display and had
> asked Tom to toggle our instance on after that model. Is the tree display
> here <https://archivesearch.lib.cam.ac.uk/repositories/2/resources/8865>
> output by the PUI configuration option we're discussing? Or does something
> else make that ID display possible?
>
> The staff-side view presented in the pull request you referenced also
> looks like exactly what we're after. The work being done, any chance this
> option could be included in the 3.4 release? Thanks again,
>
> Jennifer
>
>
>
> On Mon, May 15, 2023 at 1:41 PM Brian Hoffman 
> wrote:
>
> Hi Tom,
>
>
>
> I don’t believe it has worked in 3.3.1 or any previous release. Yes, it
> should be working in 3.5 as the work has already been done:
>
>
>
> https://github.com/archivesspace/archivesspace/pull/2987
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Monday, May 15, 2023 at 1:15 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Brian:
>
>
>
> Is this a future version such as the released version of 3.4 or a future
> version like in 3.5 or later?
>
> I'm unclear how functionality in 3.3.1 would get "lost" in the move to
> 3.4.
>
>
>
> Thanks,
>
> Tom
>
>
>
> On Mon, May 15, 2023 at 1:00 PM Brian Hoffman 
> wrote:
>
> Hi Jennifer,
>
>
>
> There is an issue with the way those features were implemented, and they
> won’t work without some customization in version 3.4.0. There is a fix
> completed that will appear in a future version.

Re: [Archivesspace_Users_Group] ID display config options

[Tom Hanstra]

Thanks for the clarification and sorry for the confusion.

Again, I find myself not familiar enough with the process for ArchivesSpace
releases and development. If the work is done (as shown in the pull request
sent), what is barring this from being added into the upcoming release of
13.4? Seems like completed work could easily be added to the next release,
but I'm probably being too simplistic?

If it won't be added by 3.4, I'll look into the plugin. If someone has a
URL for it, that would be great. Or I'll try Google.

Thanks,
Tom

On Tue, May 16, 2023 at 8:49 AM Erhan Tuskan  wrote:

> Yes, what Brian thinks must be true, at IISH we use the Cambridge plugin
> too and the Component Unique Identifiers are displayed in the PUI and staff
> client as well.
>
>
>
> Erhan,
>
>
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Brian
> Hoffman
> *Sent:* dinsdag 16 mei 2023 2:27
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] ID display config options
>
>
>
> Hi Jennifer,
>
>
>
> I believe that functionality was originally developed as a plugin –
> perhaps that explains why Cambridge has it working already?
>
>
>
> Brian
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Jennifer Brcka 
> *Date: *Monday, May 15, 2023 at 4:49 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Hi Brian,
>
> For clarification -- I'd seen IDs in Cambridge's public display and had
> asked Tom to toggle our instance on after that model. Is the tree display
> here <https://archivesearch.lib.cam.ac.uk/repositories/2/resources/8865>
> output by the PUI configuration option we're discussing? Or does something
> else make that ID display possible?
>
> The staff-side view presented in the pull request you referenced also
> looks like exactly what we're after. The work being done, any chance this
> option could be included in the 3.4 release? Thanks again,
>
> Jennifer
>
>
>
> On Mon, May 15, 2023 at 1:41 PM Brian Hoffman 
> wrote:
>
> Hi Tom,
>
>
>
> I don’t believe it has worked in 3.3.1 or any previous release. Yes, it
> should be working in 3.5 as the work has already been done:
>
>
>
> https://github.com/archivesspace/archivesspace/pull/2987
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Monday, May 15, 2023 at 1:15 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Brian:
>
>
>
> Is this a future version such as the released version of 3.4 or a future
> version like in 3.5 or later?
>
> I'm unclear how functionality in 3.3.1 would get "lost" in the move to
> 3.4.
>
>
>
> Thanks,
>
> Tom
>
>
>
> On Mon, May 15, 2023 at 1:00 PM Brian Hoffman 
> wrote:
>
> Hi Jennifer,
>
>
>
> There is an issue with the way those features were implemented, and they
> won’t work without some customization in version 3.4.0. There is a fix
> completed that will appear in a future version.
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Andrew Morrison 
> *Date: *Monday, May 15, 2023 at 12:27 PM
> *To: *archivesspace_users_group@lyralists.lyrasis.org <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Those two features are partially implemented in JavaScript. It is possible
> that your web browser has cached the old versions of the JavaScript files.
> Try doing a hard refresh
> <https://www.howtogeek.com/672607/how-to-hard-refresh-your-web-browser-to-bypass-your-cache/>
> or emptying your browser's cache.
>
> Andrew.
>
>
>
> On 15/05/2023 16:53, Jennifer Brcka wrote:
>
> Hi All,
>
>
>
> We've been testing 3.4.0-RC1 and haven't been able to get two
> configuration options to work:
>
>
> AppConfig[:display_identifiers_in_largetree_container] = true
>
> AppConfig[:pui_display_identifiers_in_re

Re: [Archivesspace_Users_Group] ID display config options

[Tom Hanstra]

Brian:

Is this a future version such as the released version of 3.4 or a future
version like in 3.5 or later?

I'm unclear how functionality in 3.3.1 would get "lost" in the move to 3.4.

Thanks,
Tom

On Mon, May 15, 2023 at 1:00 PM Brian Hoffman 
wrote:

> Hi Jennifer,
>
>
>
> There is an issue with the way those features were implemented, and they
> won’t work without some customization in version 3.4.0. There is a fix
> completed that will appear in a future version.
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Andrew Morrison 
> *Date: *Monday, May 15, 2023 at 12:27 PM
> *To: *archivesspace_users_group@lyralists.lyrasis.org <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] ID display config options
>
> Those two features are partially implemented in JavaScript. It is possible
> that your web browser has cached the old versions of the JavaScript files.
> Try doing a hard refresh
> <https://www.howtogeek.com/672607/how-to-hard-refresh-your-web-browser-to-bypass-your-cache/>
> or emptying your browser's cache.
>
> Andrew.
>
>
>
> On 15/05/2023 16:53, Jennifer Brcka wrote:
>
> Hi All,
>
>
>
> We've been testing 3.4.0-RC1 and haven't been able to get two
> configuration options to work:
>
>
> AppConfig[:display_identifiers_in_largetree_container] = true
>
> AppConfig[:pui_display_identifiers_in_resource_tree] = true
>
>
>
> Our database was reindexed after switching these to 'true.'  Any thoughts
> on what could be going on? Is there anything else we need to change to get
> this to work?
>
>
>
> Thanks,
>
> Jennifer
>
>
>
> --
>
> *Jennifer Brcka*
>
> *Archives Specialist*
> Archival Collections & Management, Hesburgh Libraries
> University of Notre Dame
>
> e: jbr...@nd.edu
>
> o: (574)631-2008
>
>
>
> ___
>
> Archivesspace_Users_Group mailing list
>
> Archivesspace_Users_Group@lyralists.lyrasis.org
>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Looking for advice: moving from 2.7.1 to 3.3.1 on a new server

[Tom Hanstra]

If this is going to become a production service, then you would want to get
a MySQL instance running and get the database built there. That software
could run locally on the server or be elsewhere, but you should move to a
full fledged database server pretty early in the process.

See:

https://archivesspace.github.io/tech-docs/provisioning/mysql.html

Tom

On Wed, Apr 26, 2023 at 3:09 PM Gadsby, Eric T.  wrote:

> Tom,
>
>
>
> Hi. Thanks!  In your process have you gotten the new Aspace running
> against Mysql/Maria on an fresh DB before importing the DB or do you just
> run it off the provided db server until then? Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>
>
>
>
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Friday, April 21, 2023 at 1:58 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] Looking for advice: moving
> from 2.7.1 to 3.3.1 on a new server
>
> [ *CAUTION*: This email is from outside of TU. Use caution before
> clicking links or opening attachments. If suspicious, report to
> phish...@towson.edu. ]
>
> Eric,
>
>
>
> I've done this a couple of times now and here is my process:
>
> - Create the server with RHEL8 and get it working as you want (any
> internal updates)
>
> - Add in the ArchivesSpace software and Solr
>
> - Get Solr running with your local configuration changes (if any)
>
> - Get ArchivesSpace running with your local changes and configuration
>
> - Add in any plugins at the same time (including SSO)
>
>
>
> At that point, I would suggest that you *test* moving over the database.
> Copy the database over to your new server, run it through the upgrade
> process, and then make sure indexing works. Test that everything is working
> as you want it. As long as the names are different between the new and old,
> you can test as much as you want.
>
> Once you know everything is working as desired/expected, *then* do the
> production copy/index processing and cut over DNS.
>
>
>
> The testing could take a while. There are quite a few changes
> between 2.7.1 and 3.3.1, not the least of which is getting Solr running
> separately.
>
>
>
> Hope that helps,
>
> Tom
>
>
>
> On Fri, Apr 21, 2023 at 1:02 PM Gadsby, Eric T. 
> wrote:
>
> Dear Friends,
>
>
>
> Our install is at 2.7.1 on RHEL 7. The pandemic contributed to a pause in
> updates.  Now that we are back to “normal” we need to move to the latest
> version on a REHL 8.6 server.   I am trying to work out the order in which
> do things to get the new server up and the data moved.  I know we’ll need
> to set up the application and Solr on the new server.  What I am trying to
> puzzle out is the best way to move the data. We can do a database export
> from the old server and then import it into the new server. The question I
> have is this, how do we go about the database updates. Do we need to update
> the old server and run the script and then export the data or can we move
> the data as is and then run the updater script in the new environment? Any
> insight would be invaluable.
>
>
>
>  I know we will also need to reestablish our SSO, reimplement plug-ins,
> and then eventually cut over our DNS. Is there something I’m not thinking
> about?  Thank you in advance for any help the community can offer. Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
&

Re: [Archivesspace_Users_Group] Looking for advice: moving from 2.7.1 to 3.3.1 on a new server

[Tom Hanstra]

Eric,

I've done this a couple of times now and here is my process:

- Create the server with RHEL8 and get it working as you want (any internal
updates)
- Add in the ArchivesSpace software and Solr
- Get Solr running with your local configuration changes (if any)
- Get ArchivesSpace running with your local changes and configuration
- Add in any plugins at the same time (including SSO)

At that point, I would suggest that you *test* moving over the database.
Copy the database over to your new server, run it through the upgrade
process, and then make sure indexing works. Test that everything is working
as you want it. As long as the names are different between the new and old,
you can test as much as you want.

Once you know everything is working as desired/expected, *then* do the
production copy/index processing and cut over DNS.

The testing could take a while. There are quite a few changes between 2.7.1
and 3.3.1, not the least of which is getting Solr running separately.

Hope that helps,
Tom

On Fri, Apr 21, 2023 at 1:02 PM Gadsby, Eric T.  wrote:

> Dear Friends,
>
>
>
> Our install is at 2.7.1 on RHEL 7. The pandemic contributed to a pause in
> updates.  Now that we are back to “normal” we need to move to the latest
> version on a REHL 8.6 server.   I am trying to work out the order in which
> do things to get the new server up and the data moved.  I know we’ll need
> to set up the application and Solr on the new server.  What I am trying to
> puzzle out is the best way to move the data. We can do a database export
> from the old server and then import it into the new server. The question I
> have is this, how do we go about the database updates. Do we need to update
> the old server and run the script and then export the data or can we move
> the data as is and then run the updater script in the new environment? Any
> insight would be invaluable.
>
>
>
>  I know we will also need to reestablish our SSO, reimplement plug-ins,
> and then eventually cut over our DNS. Is there something I’m not thinking
> about?  Thank you in advance for any help the community can offer. Thanks!
>
>
>
>
>
>
>
> [image: Towson University logo] <http://www.towson.edu/>
>
> *Eric T. Gadsby*
>
> *Pronouns: he/him/his*
>
> IT Operations Specialist  |  Albert S. Cook Library
>
> *—*
>
> P: 410-704-3340
> egad...@towson.edu  |  libraries.towson.edu
> <http://www.towson.edu/https:/libraries.towson.edu>
>  *—*
>
>
>
> Confidentiality Notice: This message may contain information that is
> confidential, privileged, proprietary, or otherwise legally exempt from
> disclosure. If you are not the intended recipient, you are notified that
> you are not authorized to read, print, copy or disseminate this message,
> any part of it, or any attachments. If this message has been sent to you in
> error, please notify the sender by replying to this transmission, or by
> calling Albert S. Cook Library at 410-704-3340 .
>
>
>
>
> _______
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] ASpace database issues

[Tom Hanstra]

Is there any way that your version of MySQL got upgraded?  I see this in
the MySQL documentation:

https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8.html

which indicates that utf8mb3 is deprecated after Mysql 8.0.28.

I don't have any experience to say that this is where the problem is. I was
just looking around because I don't want to end up in a similar situation.

Tom

On Wed, Jan 18, 2023 at 2:23 PM Matthew Adair  wrote:

> Version 2.5.2 of ASpace.  Version 8.0.30 of MySQL
>
> We get the following error in the log file:
> "The following MySQL database tables are not set to use UTF-8 for their
> character encoding:"
> and then it lists all of the database tables, and fails with an unable to
> connect to database error
>
> Encoding on the database and all of the tables are utf8mb3_general_ci
>
> This problem seems to have just cropped up and there have been no
> changes that I am aware of.
>
> Thoughts?
> -Matt
>
> 
> *Matt Adair*
> Archivist for Digital Imaging and Infrastructure
>
>
> Bentley Historical Library
> 1150 Beal Avenue
> Ann Arbor, Michigan 48109-2113
> 734-647-3537
> http://bentley.umich.edu
> @UmichBentley
>
> *The Bentley Historical Library acknowledges that coerced cessions of land
> by the Anishnaabeg and Wyandot made the University of Michigan possible,
> and we seek to reaffirm the ancestral and contemporary ties of these
> peoples to the lands where the University now stands.*
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Startup error with ArchivesSpace v3.3.1/aspace-oauth plugin

[Tom Hanstra]

Try updating the Gemfile.lock to use the 4.0.6 version of the gem and then
run the initialize_plugin script. I'm pretty sure that was how I got around
the issue at one point.

You just need to get the plugin bundled. Version does not matter (in my
experience).

Tom

On Tue, Dec 6, 2022 at 3:03 PM David P. Steelman  wrote:

> Thanks for the quick responses.
>
> I have tried using aspace-oauth v3.2.0, and am running the
> 'initialize_plugin" script, and still receiving the same error.
>
> On the server, there is a
> "plugins/aspace-oauth/gems/cache/public_suffix-4.0.7.gem" file, and a
> "plugins/aspace-oauth/gems/gems/public_suffix-4.0.7" directory, so the
> "aspace-oauth" gem appears to be using the "public_suffix:4.0.7" gem
>
> The "WEB-INF/Gemfile.lock" file in the "public.war" and "frontend.war" WAR
> files from the ArchivesSpace distribution each contain a reference to
> "public_suffix" v4.0.6.
>
> In the ArchivesSpace v3.3.1 source distribution there are references to
> "plugin_suffix:4.0.6" in the following files:
>
> * Gemfile.lock
> * docs/Gemfile.lock
> * frontend/Gemfile.lock
> * public/Gemfile.lock
>
> So I'm wondering if there is some kind of confusion about which version of
> the gem is supposed to be used.
>
> Thanks,
>
> David
>
> On Tue, Dec 6, 2022 at 2:55 PM Tom Hanstra  wrote:
>
>> There was a version of the aspace-oauth plugin in which the public_suffix
>> gem did not compile. I remember having some problems with that.
>>
>> I just rebuilt mine here and it looks like the current version is working
>> right, with the 4.0.7 version. So it may help to simply download again.
>>
>> Tom
>>
>> On Tue, Dec 6, 2022 at 2:19 PM Blake Carver 
>> wrote:
>>
>>> Try using the latest tag version on that plugin.
>>>
>>>
>>>  https://github.com/lyrasis/aspace-oauth/tags
>>> --
>>> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
>>> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
>>> David P. Steelman 
>>> *Sent:* Tuesday, December 6, 2022 2:02 PM
>>> *To:* Archivesspace Users Group <
>>> archivesspace_users_group@lyralists.lyrasis.org>
>>> *Subject:* [Archivesspace_Users_Group] Startup error with ArchivesSpace
>>> v3.3.1/aspace-oauth plugin
>>>
>>> Hello all,
>>>
>>> In attempting to upgrade from ArchivesSpace v3.1.1 to v3.3.1, I am
>>> encountering the following error on startup:
>>>
>>> 
>>>   Dec 05, 2022 8:23:43 PM
>>> org.eclipse.jetty.server.handler.ContextHandler$Context log
>>>   WARNING: ERROR: initialization failed
>>>   org.jruby.rack.RackInitializationException: Could not find
>>> public_suffix-4.0.6 in any of the sources
>>>   from
>>> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in
>>> `block in materialize'
>>>   from org/jruby/RubyArray.java:2621:in `map!'
>>>   from
>>> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in
>>> `materialize'
>>>   from
>>> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in
>>> `specs'
>>>   ...
>>> 
>>>
>>> We are using the "aspace-oauth" plugin (
>>> https://github.com/lyrasis/aspace-oauth) -- if we remove this plugin
>>> from the configuration the error does not occur.
>>>
>>> Comparing the ArchivesSpace v3.1.1 distribution to v3.3.1, the
>>> "gems/gems" directory in the v3.3.1 distribution includes significantly
>>> fewer gems, and no longer includes a "public_suffix-3.0.6" directory.
>>>
>>> Any pointers in handling this issue are appreciated.
>>>
>>> Thanks,
>>>
>>> David
>>> ___
>>> Archivesspace_Users_Group mailing list
>>> Archivesspace_Users_Group@lyralists.lyrasis.org
>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>>
>>
>>
>> --
>> *Tom Hanstra*
>> *Sr. Systems Administrator*
>> hans...@nd.edu
>>
>>
>> ___
>> Archivesspace_Users_Group mailing list
>> Archivesspace_Users_Group@lyralists.lyrasis.org
>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Startup error with ArchivesSpace v3.3.1/aspace-oauth plugin

[Tom Hanstra]

There was a version of the aspace-oauth plugin in which the public_suffix
gem did not compile. I remember having some problems with that.

I just rebuilt mine here and it looks like the current version is working
right, with the 4.0.7 version. So it may help to simply download again.

Tom

On Tue, Dec 6, 2022 at 2:19 PM Blake Carver 
wrote:

> Try using the latest tag version on that plugin.
>
>
>  https://github.com/lyrasis/aspace-oauth/tags
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> David P. Steelman 
> *Sent:* Tuesday, December 6, 2022 2:02 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] Startup error with ArchivesSpace
> v3.3.1/aspace-oauth plugin
>
> Hello all,
>
> In attempting to upgrade from ArchivesSpace v3.1.1 to v3.3.1, I am
> encountering the following error on startup:
>
> 
>   Dec 05, 2022 8:23:43 PM
> org.eclipse.jetty.server.handler.ContextHandler$Context log
>   WARNING: ERROR: initialization failed
>   org.jruby.rack.RackInitializationException: Could not find
> public_suffix-4.0.6 in any of the sources
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in
> `block in materialize'
>   from org/jruby/RubyArray.java:2621:in `map!'
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in
> `materialize'
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in
> `specs'
>   ...
> 
>
> We are using the "aspace-oauth" plugin (
> https://github.com/lyrasis/aspace-oauth) -- if we remove this plugin from
> the configuration the error does not occur.
>
> Comparing the ArchivesSpace v3.1.1 distribution to v3.3.1, the "gems/gems"
> directory in the v3.3.1 distribution includes significantly fewer gems, and
> no longer includes a "public_suffix-3.0.6" directory.
>
> Any pointers in handling this issue are appreciated.
>
> Thanks,
>
> David
> _______
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Startup error with ArchivesSpace v3.3.1/aspace-oauth plugin

[Tom Hanstra]

Be sure to run the initialize_plugin script again after the upgrade.

Tom

On Tue, Dec 6, 2022 at 2:06 PM David P. Steelman  wrote:

> Hello all,
>
> In attempting to upgrade from ArchivesSpace v3.1.1 to v3.3.1, I am
> encountering the following error on startup:
>
> 
>   Dec 05, 2022 8:23:43 PM
> org.eclipse.jetty.server.handler.ContextHandler$Context log
>   WARNING: ERROR: initialization failed
>   org.jruby.rack.RackInitializationException: Could not find
> public_suffix-4.0.6 in any of the sources
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in
> `block in materialize'
>   from org/jruby/RubyArray.java:2621:in `map!'
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in
> `materialize'
>   from
> /apps/aspace/archivesspace/gems/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in
> `specs'
>   ...
> 
>
> We are using the "aspace-oauth" plugin (
> https://github.com/lyrasis/aspace-oauth) -- if we remove this plugin from
> the configuration the error does not occur.
>
> Comparing the ArchivesSpace v3.1.1 distribution to v3.3.1, the "gems/gems"
> directory in the v3.3.1 distribution includes significantly fewer gems, and
> no longer includes a "public_suffix-3.0.6" directory.
>
> Any pointers in handling this issue are appreciated.
>
> Thanks,
>
> David
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] API and OAI slowdowns in 3.x

[Tom Hanstra]

In the testing which we have done with both ArchivesSpace 3.2 and 3.3, we
are seeing significant slowdowns in both OAI and API processing.

I'm trying to determine if this is something we have misconfigured on our
local servers or if this is something inherent to the newer versions of
ArchivesSpace. We initially noticed it mostly with large records, but
additional testing shows that it is happening in general. Our platform for
testing has similar resources (memory, processor) as our 2.8.1 version, but
exports are taking much longer than expected.

My initial thought was that the external Solr might be at issue, but
searches in the PUI seem to work as expected and are not overly slow. Also
looked at our database instance, but again that is similar to our
production service and seems to be working ok. So I'm at a bit of a loss as
to why exports are showing this slow response.

Are others seeing anything like this?  Any thoughts of other places to look
for issues?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Docker build

[Tom Hanstra]

Thanks, Megan and Joshua.

I do see that the ArchivesSpace repository includes some Docker build files
so I was hoping to look at that first. Unfortunately, what I cannot seem to
track down is documentation on how that exactly works. Has anyone used the
images and compose files in the repository who might be able to help me
better understand what is there and how it can be used?

In general, are there sites using Docker deployments in production?  I'm
surprised by the lack of response on this.

Tom

On Tue, Aug 2, 2022 at 9:13 PM Schanz, Megan  wrote:

> Hi Tom,
>
> This is our setup:
> https://gitlab.msu.edu/msu-libraries/public/archivesspace-docker that
> uses separate Docker containers for ArchivesSpace and Solr, running v3.1.1.
>
> - Megan
>
>
>
> _
>
> Megan Schanz
> Application Developer & Systems Administrator
> Michigan State University Libraries
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> Joshua D. Shaw 
> *Sent:* Tuesday, August 2, 2022 4:17 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Docker build
>
> Hey Tom
>
> Here's a repo describing the setup we use:
> https://github.com/dartmouth-dltg/aspace-docker
> <https://urldefense.com/v3/__https://github.com/dartmouth-dltg/aspace-docker__;!!HXCxUKc!yil8T2g2wVSd4ySUioWNJ0I1VxqQYmBScBSnM6glmOYMpwGgNHIWOmVBwucHKjWxbPYb4WrTAafOfclP19Q5IOZazm0o$>
>
> This is *pre*​ AS v3.2.0, so no external solr - we're still running AS
> v3.1.1 in prooduction.
>
> jds
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Tuesday, August 2, 2022 1:58 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] Docker build
>
> I would like to look at running ArchivesSpace within a Docker image. I see
> at least a couple of pre-built containers available in DockerHub. Is the
> ArchivesSpace version the preferred version or are others better?  And is
> there good documentation somewhere on how to run AS within a docker
> container, including proper setup instructions?
>
> Thanks for any help you can provide to get me started,
> Tom
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Docker build

[Tom Hanstra]

I would like to look at running ArchivesSpace within a Docker image. I see
at least a couple of pre-built containers available in DockerHub. Is the
ArchivesSpace version the preferred version or are others better?  And is
there good documentation somewhere on how to run AS within a docker
container, including proper setup instructions?

Thanks for any help you can provide to get me started,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Setting AS and Solr to just keep running?

[Tom Hanstra]

Few questions related to Linux are "super basic". And most, including this
one, are answered with "it depends".

If you can contact me directly with info on the version of Linux and how
you currently start solr and archivesspace, I can probably help you get
things set up as you are hoping.

Tom

On Fri, Jun 10, 2022 at 11:48 AM Sarah Frieldsmith <
sfrieldsm...@sandiego.edu> wrote:

> I know this is a super basic question and more related to my lack of linux
> knowledge than AS itself
>
> How do I get archivesspace and solr to just keep running? I can start them
> both manually. I did ask our server guys and they pointed to a github
> solution and, well, I am still lost.
>
> Singing "just keep running, just keep running" isn't working :P
>
> *SARAH FRIELDSMITH*
> *Systems Librarian*
> University of San Diego
> 5998 Alcalá Park
> San Diego, CA 92110-2492
> (619) 260-4776
> sfrieldsm...@sandiego.edu
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Problems with oauth plugin

[Tom Hanstra]

Blake and Peter,

Thanks for pointing me in the right direction. I could not see any reason
why our OKTA metadata URL would be complaining, but Blake's suggestion
allows us to bypass that piece (at least for now). Authentication through
OKTA is back up and working for us now.

Tom

On Thu, Jun 2, 2022 at 8:43 PM Blake Carver 
wrote:

> add this:
>   verify_ssl: false,
> To you config, after metadata_parser_url and above the config: {
>
> So it'll look something like this:
>
> metadata_parser_url: "https://someloginurl.example;,
> verify_ssl: false,
> config: {
>blah blah blah
>
>
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Thursday, June 2, 2022 9:37 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Problems with oauth plugin
>
> Thanks, Blake. Unfortunately, that did not do it. The install script works
> but we still get this complaint about the certificate verification:
>
> I'm attaching the entire error as a separate file. Perhaps someone with
> more Ruby understanding will see something in there that I have not. If I
> could figure out what certificate/file it is looking at, perhaps I could
> track this down. Or maybe it is a red herring and there is something else
> going on in there.
>
> Tom
>
> On Wed, Jun 1, 2022 at 5:10 PM Blake Carver 
> wrote:
>
> You might try this branch, there was a weird issue with that for a while,
> I think maybe this fixed that?
> https://github.com/lyrasis/aspace-oauth/tree/unlock-address
>
> This was the only change
> https://github.com/lyrasis/aspace-oauth/pull/23/files
>
> That was a while back, so things may have changed since on some of those
> gems.
> ----------
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Wednesday, June 1, 2022 2:22 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] Problems with oauth plugin
>
> I'm having some problems with our Authentication with OKTA which I'm
> trying to understand.
>
> Because of the problems, I've tried reinstalling the oauth plugin
> completely. The first problem I ran into was that the current download of:
>
> https://github.com/lyrasis/aspace-oauth.git
>
> Had a Gemfile containing the line:
>
> gem 'addressable',   '2.8.0'
>
> This caused some gem issues with our 2.81. version of ArchivesSpace
> because 2.8.0 was evidently newer than the 2.7.0 version that is in the
> gems directory. I'm not savvy enough with Ruby to know how to deal with
> that so I simply updated the aspace-oauth Gemvile to read:
>
> gem 'addressable',   '2.7.0'
>
> Not sure if that is legit or not. But it allowed the initialize-plugin
> script to work.
>
> But I'm still running into what was actually the original error we are
> getting. In the archivesspace.out file, we see this error:
>
> 
> INFO: An exception happened during JRuby-Rack startup
> certificate verify failed
> --- System
> jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
> 25.312-b07 on 1.8.0_312-b07 +jit [linux-x86_64]
> Time: 2022-06-01 13:57:45 -0400
> Server: jetty/8.1.5.v20120716
> jruby.home: uri:classloader://META-INF/jruby.home
>
> --- Context Init Parameters:
> jruby.max.runtimes = 1
> jruby.min.runtimes = 1
> public.root = /
> rails.env = production
>
> --- Backtrace
> OpenSSL::SSL::SSLError: certificate verify failed
>  connect at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002
> do_start at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924
>start at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913
>  request at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465
>
> [and a lot more ruby stuff]
> --
>
> There seems to be some certificate that the plugin is not happy about. But
> I cannot determine what certificate it does not like. Both the local
> certificates and the OKTA certificates are valid. So what is the issue?
>
> Anyone seen this before and have ideas?
>
> Thanks,
> Tom
>
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans

Re: [Archivesspace_Users_Group] Problems with oauth plugin

[Tom Hanstra]

Thanks, Blake. Unfortunately, that did not do it. The install script works
but we still get this complaint about the certificate verification:

I'm attaching the entire error as a separate file. Perhaps someone with
more Ruby understanding will see something in there that I have not. If I
could figure out what certificate/file it is looking at, perhaps I could
track this down. Or maybe it is a red herring and there is something else
going on in there.

Tom

On Wed, Jun 1, 2022 at 5:10 PM Blake Carver 
wrote:

> You might try this branch, there was a weird issue with that for a while,
> I think maybe this fixed that?
> https://github.com/lyrasis/aspace-oauth/tree/unlock-address
>
> This was the only change
> https://github.com/lyrasis/aspace-oauth/pull/23/files
>
> That was a while back, so things may have changed since on some of those
> gems.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Wednesday, June 1, 2022 2:22 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] Problems with oauth plugin
>
> I'm having some problems with our Authentication with OKTA which I'm
> trying to understand.
>
> Because of the problems, I've tried reinstalling the oauth plugin
> completely. The first problem I ran into was that the current download of:
>
> https://github.com/lyrasis/aspace-oauth.git
>
> Had a Gemfile containing the line:
>
> gem 'addressable',   '2.8.0'
>
> This caused some gem issues with our 2.81. version of ArchivesSpace
> because 2.8.0 was evidently newer than the 2.7.0 version that is in the
> gems directory. I'm not savvy enough with Ruby to know how to deal with
> that so I simply updated the aspace-oauth Gemvile to read:
>
> gem 'addressable',   '2.7.0'
>
> Not sure if that is legit or not. But it allowed the initialize-plugin
> script to work.
>
> But I'm still running into what was actually the original error we are
> getting. In the archivesspace.out file, we see this error:
>
> 
> INFO: An exception happened during JRuby-Rack startup
> certificate verify failed
> --- System
> jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
> 25.312-b07 on 1.8.0_312-b07 +jit [linux-x86_64]
> Time: 2022-06-01 13:57:45 -0400
> Server: jetty/8.1.5.v20120716
> jruby.home: uri:classloader://META-INF/jruby.home
>
> --- Context Init Parameters:
> jruby.max.runtimes = 1
> jruby.min.runtimes = 1
> public.root = /
> rails.env = production
>
> --- Backtrace
> OpenSSL::SSL::SSLError: certificate verify failed
>  connect at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002
> do_start at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924
>start at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913
>  request at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465
>
> [and a lot more ruby stuff]
> --
>
> There seems to be some certificate that the plugin is not happy about. But
> I cannot determine what certificate it does not like. Both the local
> certificates and the OKTA certificates are valid. So what is the issue?
>
> Anyone seen this before and have ideas?
>
> Thanks,
> Tom
>
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
servlet_context = 
ServletContext@o.e.j.w.WebAppContext{/,file:/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8080-frontend.war-_-any-/webapp/},/home/app/archivesspace/wars/frontend.war
throw_init_exception = false

Jun 02, 2022 9:29:30 AM org.eclipse.jetty.server.handler.ContextHandler$Context 
log
WARNING: ERROR: initialization failed
org.jruby.rack.RackInitializationException: certificate verify failed
from 
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002:in 
`connect'
from 
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in 
`do_start'
from 
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'
from 
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465:in 
`request'
from 
/home/app/archivesspace/plugins/aspace

[Archivesspace_Users_Group] Problems with oauth plugin

[Tom Hanstra]

I'm having some problems with our Authentication with OKTA which I'm trying
to understand.

Because of the problems, I've tried reinstalling the oauth plugin
completely. The first problem I ran into was that the current download of:

https://github.com/lyrasis/aspace-oauth.git

Had a Gemfile containing the line:

gem 'addressable',   '2.8.0'

This caused some gem issues with our 2.81. version of ArchivesSpace because
2.8.0 was evidently newer than the 2.7.0 version that is in the gems
directory. I'm not savvy enough with Ruby to know how to deal with that so
I simply updated the aspace-oauth Gemvile to read:

gem 'addressable',   '2.7.0'

Not sure if that is legit or not. But it allowed the initialize-plugin
script to work.

But I'm still running into what was actually the original error we are
getting. In the archivesspace.out file, we see this error:


INFO: An exception happened during JRuby-Rack startup
certificate verify failed
--- System
jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
25.312-b07 on 1.8.0_312-b07 +jit [linux-x86_64]
Time: 2022-06-01 13:57:45 -0400
Server: jetty/8.1.5.v20120716
jruby.home: uri:classloader://META-INF/jruby.home

--- Context Init Parameters:
jruby.max.runtimes = 1
jruby.min.runtimes = 1
public.root = /
rails.env = production

--- Backtrace
OpenSSL::SSL::SSLError: certificate verify failed
 connect at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002
do_start at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924
   start at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913
 request at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1465

[and a lot more ruby stuff]
--

There seems to be some certificate that the plugin is not happy about. But
I cannot determine what certificate it does not like. Both the local
certificates and the OKTA certificates are valid. So what is the issue?

Anyone seen this before and have ideas?

Thanks,
Tom


--
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

Thanks again.

So, I currently break up my logs into four different individual logs with
these settings:

## Possible Log levels: debug, info, warn, error, fatal (severe only)
#
AppConfig[:frontend_log] = "/var/log/archivesspace/frontend.log"
AppConfig[:frontend_log_level] = "debug"
#
AppConfig[:backend_log] = "/var/log/archivesspace/backend.log"
AppConfig[:backend_log_level] = "debug"
#
AppConfig[:pui_log] = "/var/log/archivesspace/pui.log"
AppConfig[:pui_log_level] = "debug"
#
AppConfig[:indexer_log] = "/var/log/archivesspace/indexer.log"
AppConfig[:indexer_log_level] = "debug"

In each of the other logs I see both INFO and DEBUG statements, sometimes a
lot, other times fewer. But in the indexer.log file, I only see INFO
statements, no DEBUG.

But your suggestion above that the line is like "indexed ..." got me
searching other logs and I found a couple of lines like this in my backend
log:

D, [2022-05-16T14:31:45.734159 #5273] DEBUG -- : Thread-4912: Responded
with [200, {"Content-Type"=>"application/json", "Cache-Control"=>"private,
must-revalidate, max-age=0", "Content-Length"=>"21295"},
["{\"page_size\":1,\"first_page\":1,\"last_page\":1,\"this_page\":1,\"offset_first\":1,\"offset_last\":1,\"total_hits\":1,\"results\":[{\"id\":\"/repositories/2/archival_objects/1702566#pui\",\"uri\":\"/repositories/2/archival_objects/1702566\",\"title\":\"Brother
Paul Hermit CSC not
indexed.\",\"primary_type\":\"archival_object\",\"types\":[\"archival_object\",\"pui\",\"pui_archival...
in 73ms

Is this the type of thing I should continue to look for?

Thanks,
Tom

On Wed, May 18, 2022 at 4:12 PM Blake Carver 
wrote:

> > - Even though the config setting is for DEBUG, the indexer log is only
> showing INFO. Should there be more than that?
>
> Yep. Something must not be quite right in the config, you should see DEBUG
> showing up in the logs.
>
> If it's running 1x1 AND on DEBUG you should see something better by the
> crash, you should be able to spot the record. I can't remember what exactly
> it looks like, but something like "indexed some/path/with/an/id/in/it/here"
> and then you can see the Resource or Archival Object ID in the line there.
>
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Wednesday, May 18, 2022 3:21 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> Blake and others,
>
> Overall, we went for a week or so with no errors in terms of re-indexing.
> I'm looking into some database timeouts which are occurring periodically
> (AS and our database server are in two different locations) but with the
> connection getting re-established, I can work to address that separately.
>
> But last night we did run into some errors which we are trying to track
> down.  A couple of things:
>
> - Even though the config setting is for DEBUG, the indexer log is only
> showing INFO. Should there be more than that?
>
> - If the log says:
>
> I, [2022-05-17T22:27:11.756683 #5273]  INFO -- : Thread-2890: Staff
> Indexer [2022-05-17 22:27:11 -0400] ~~~ Indexed 216
> of 20898 archival_object records in repository UNDA
>
> immediately followed by an error, how do we determine which record that is
> in ArchivesSpace?  We are not sure how to find the record which is
> indicated in the log.
>
> Thanks again,
> Tom
>
>
>
> On Mon, May 16, 2022 at 5:19 PM Blake Carver 
> wrote:
>
> That could be it, looks like some kind of "funny" character ArchivesSpace
> doesn't like in there probably. Do the "Indexed x of x whatever " numbers
> start over right there in the log?
> You should see what record it was working on just before there.
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Monday, May 16, 2022 10:43 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> I've been monitoring logs and have run into a couple of instances where I
> receive this type of error:
>
> E, [2022-05-16T08:57:57.207392 #5273] ERRO

Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

Blake and others,

Overall, we went for a week or so with no errors in terms of re-indexing.
I'm looking into some database timeouts which are occurring periodically
(AS and our database server are in two different locations) but with the
connection getting re-established, I can work to address that separately.

But last night we did run into some errors which we are trying to track
down.  A couple of things:

- Even though the config setting is for DEBUG, the indexer log is only
showing INFO. Should there be more than that?

- If the log says:

I, [2022-05-17T22:27:11.756683 #5273]  INFO -- : Thread-2890: Staff Indexer
[2022-05-17 22:27:11 -0400] ~~~ Indexed 216
of 20898 archival_object records in repository UNDA

immediately followed by an error, how do we determine which record that is
in ArchivesSpace?  We are not sure how to find the record which is
indicated in the log.

Thanks again,
Tom



On Mon, May 16, 2022 at 5:19 PM Blake Carver 
wrote:

> That could be it, looks like some kind of "funny" character ArchivesSpace
> doesn't like in there probably. Do the "Indexed x of x whatever " numbers
> start over right there in the log?
> You should see what record it was working on just before there.
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Monday, May 16, 2022 10:43 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> I've been monitoring logs and have run into a couple of instances where I
> receive this type of error:
>
> E, [2022-05-16T08:57:57.207392 #5273] ERROR -- : Thread-3962: Unhandled
> exception!
> E, [2022-05-16T08:57:57.207810 #5273] ERROR -- :
> invalid byte sequence in UTF-8
>
> and then a lot of detail. Are these errors with data that should be
> addressed?  I'm still in DEBUG mode in the backend log but it is not clear
> how to figure out which record(s) are being flagged. What am I looking for?
>
> Thanks,
> Tom
>
> On Thu, May 12, 2022 at 8:59 AM Blake Carver 
> wrote:
>
> > Could it be that we added a big record which is now having issues being
> extracted. Or a corrupted record which is causing such issues?
>
>  I don't think so? It could be, but those errors make me think there's
> something else going on with the DB, or maybe the network between the
> application and DB server? I'm not really sure what the problem is, but it
> seems more like a hardware/network/server issue than an ArchivesSpace
> issue. I can't be sure, but those errors don't look like ArchivesSpace
> troubles to me. Those are pretty common errors, so I'd do some searching
> around to see what you can find to troubleshoot.
> --------------
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Wednesday, May 11, 2022 9:30 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> Thanks again for your help.
>
> Late yesterday, both indexes indicated completion so I thought maybe
> things were going to be OK. Consequently, I did not do much in terms of
> testing.
>
> This morning, the logs again had errors, however.  In the logs, I found
> this error in the indexer log:
>
> I, [2022-05-10T21:36:07.181427 #30003]  INFO -- : Thread-2890: Staff
> Indexer [2022-05-10 21:36:07 -0400] Index round complete
> I, [2022-05-10T21:36:37.182006 #30003]  INFO -- : Thread-2890: Staff
> Indexer [2022-05-10 21:36:37 -0400] Running index round
> E, [2022-05-10T21:36:37.283423 #30003] ERROR -- : Thread-2890:
> uri:classloader:/jsonmodel_client.rb:493:in `all'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:154:in
> `run_index_round'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:283:in
> `run'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/main.rb:32:in
> `block in main'
> E, [2022-05-10T21:36:37.284431 #30003] ERROR -- : Thread-2890:
> # conflict: Java::JavaSql::SQLNonTransientConnectionException: No operations
> allowed after connection closed."]}}
>
> and in the backup log there were issues with timeouts retrieving a record:
>
> Java::ComMysqlCjJdbcExceptions::CommunicationsException: The last packet
> successfully received from

Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

I've been monitoring logs and have run into a couple of instances where I
receive this type of error:

E, [2022-05-16T08:57:57.207392 #5273] ERROR -- : Thread-3962: Unhandled
exception!
E, [2022-05-16T08:57:57.207810 #5273] ERROR -- :
invalid byte sequence in UTF-8

and then a lot of detail. Are these errors with data that should be
addressed?  I'm still in DEBUG mode in the backend log but it is not clear
how to figure out which record(s) are being flagged. What am I looking for?

Thanks,
Tom

On Thu, May 12, 2022 at 8:59 AM Blake Carver 
wrote:

> > Could it be that we added a big record which is now having issues being
> extracted. Or a corrupted record which is causing such issues?
>
>  I don't think so? It could be, but those errors make me think there's
> something else going on with the DB, or maybe the network between the
> application and DB server? I'm not really sure what the problem is, but it
> seems more like a hardware/network/server issue than an ArchivesSpace
> issue. I can't be sure, but those errors don't look like ArchivesSpace
> troubles to me. Those are pretty common errors, so I'd do some searching
> around to see what you can find to troubleshoot.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Wednesday, May 11, 2022 9:30 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> Thanks again for your help.
>
> Late yesterday, both indexes indicated completion so I thought maybe
> things were going to be OK. Consequently, I did not do much in terms of
> testing.
>
> This morning, the logs again had errors, however.  In the logs, I found
> this error in the indexer log:
>
> I, [2022-05-10T21:36:07.181427 #30003]  INFO -- : Thread-2890: Staff
> Indexer [2022-05-10 21:36:07 -0400] Index round complete
> I, [2022-05-10T21:36:37.182006 #30003]  INFO -- : Thread-2890: Staff
> Indexer [2022-05-10 21:36:37 -0400] Running index round
> E, [2022-05-10T21:36:37.283423 #30003] ERROR -- : Thread-2890:
> uri:classloader:/jsonmodel_client.rb:493:in `all'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:154:in
> `run_index_round'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:283:in
> `run'
> /home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/main.rb:32:in
> `block in main'
> E, [2022-05-10T21:36:37.284431 #30003] ERROR -- : Thread-2890:
> # conflict: Java::JavaSql::SQLNonTransientConnectionException: No operations
> allowed after connection closed."]}}
>
> and in the backup log there were issues with timeouts retrieving a record:
>
> Java::ComMysqlCjJdbcExceptions::CommunicationsException: The last packet
> successfully received from the server was 1,759 milliseconds ago. The last
> packet sent successfully to the server was 28,849,143 milliseconds ago. is
> longer than the server configured value of 'wait_timeout'. You should
> consider either expiring and/or testing connection validity before use in
> your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
>
> Could it be that we added a big record which is now having issues being
> extracted. Or a corrupted record which is causing such issues?
>
> I've now restarted with the 1x1 and DEBUG on and only staff indexing and
> it is still thinking indexing is complete. I'll keep things going this way
> until we hit an error again and hopefully that will give additional
> information.
>
> I'll also look into the "autoReconnect=true" piece, since we seem to have
> a situation where, once this happens, nothing more works until a restart.
>
> Thanks again for any thoughts on this,
> Tom
>
> On Wed, May 11, 2022 at 5:03 AM Andrew Morrison <
> andrew.morri...@bodleian.ox.ac.uk> wrote:
>
> Indexing can also fail at the commit stage, not related to any one record.
> That is when ArchivesSpace tells Solr to transfer changes made in memory to
> storage. It does that at several points in the indexing process, but the
> longest one is at the end of the PUI indexer's run. If, because you've got
> a lot of records, or slow storage on your Solr server, it takes longer it
> respond than the value of AppConfig[:indexer_solr_timeout_seconds], it will
> start all over again, and potentially go into a loop. The workaround is 

Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

Thanks again for your help.

Late yesterday, both indexes indicated completion so I thought maybe things
were going to be OK. Consequently, I did not do much in terms of testing.

This morning, the logs again had errors, however.  In the logs, I found
this error in the indexer log:

I, [2022-05-10T21:36:07.181427 #30003]  INFO -- : Thread-2890: Staff
Indexer [2022-05-10 21:36:07 -0400] Index round complete
I, [2022-05-10T21:36:37.182006 #30003]  INFO -- : Thread-2890: Staff
Indexer [2022-05-10 21:36:37 -0400] Running index round
E, [2022-05-10T21:36:37.283423 #30003] ERROR -- : Thread-2890:
uri:classloader:/jsonmodel_client.rb:493:in `all'
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:154:in
`run_index_round'
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/periodic_indexer.rb:283:in
`run'
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexer.war-_aspace-indexer-any-/webapp/WEB-INF/app/main.rb:32:in
`block in main'
E, [2022-05-10T21:36:37.284431 #30003] ERROR -- : Thread-2890:
# wrote:

> Indexing can also fail at the commit stage, not related to any one record.
> That is when ArchivesSpace tells Solr to transfer changes made in memory to
> storage. It does that at several points in the indexing process, but the
> longest one is at the end of the PUI indexer's run. If, because you've got
> a lot of records, or slow storage on your Solr server, it takes longer it
> respond than the value of AppConfig[:indexer_solr_timeout_seconds], it will
> start all over again, and potentially go into a loop. The workaround is to
> increase the timeout.
>
>
> You might not notice you've got enough records to cause this until you do
> a full re-index, or someone edits something linked to most or all records
> (e.g. a repository, or a very widely-used subject), triggering the
> re-indexing of most of the system's records.
>
>
> Andrew.
>
>
>
> On 10/05/2022 22:06, Blake Carver wrote:
>
>  1x1 would mean setting both records_per_thread and thread_count to 1.
> Having loglevel on debug and running at 1x1, you'll be able to see exactly
> which thing is being indexed as it happens, and when it crashes, you'll see
> what it was working through at the time.
>
> PUI will always take longer, and a VERY long time 1x1, but unless you're
> sure which indexer is crashing, I'd switch them both up.
>
> You can just `grep Indexed archivesspace.out` after it's running and watch
> those numbers. As long as they're going up, all is well.
>
> It is also possible that it will finish without crashing running so slow
> as well. I've seen that happen with LARGE records.
> ----------
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org
> 
>  on behalf of
> Tom Hanstra  
> *Sent:* Tuesday, May 10, 2022 4:15 PM
> *To:* Archivesspace Users Group
> 
> 
> *Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> Thanks, Blake.
>
> Turns out we did add quite a few records recently, so maybe there was
> something in there that it did not like all that much.
>
> How can you tell which record it is choking on?  Is that your "1x1"
> suggestion?  Or does the DEBUG option make that more clear?  I have my
> indexing set to:
>
> AppConfig[:indexer_records_per_thread]  = 25
> AppConfig[:indexer_thread_count]= 2
>
> for both PUI and Staff records. I believe you are suggesting it would most
> easily be found using 1 and 1?  I can see where that could take a long
> time. But it if is going to choke over and over on the same record, then
> that may be the best way to address it.
>
> Do you think if I just did staff indexing without PUI, that it would be
> identified faster?  Or could it pass the staff side but then die on PUI
> later?
>
> I hope to try some of these ideas after hours today, so if you can confirm
> that I've got the right idea, that would help.
>
> Tom
>
>
> On Tue, May 10, 2022 at 2:17 PM Blake Carver 
> wrote:
>
> > Is this possible?
>
> Short answer, Yes, it's possible your indexer is starting over.
>
> Long answer. This can be tricky to figure out. Something is wrong, the
> indexer never wants to do that. Sometimes "something" "bad" gets into
> ArchivesSpace and the indexer will just crash and start over. The problem
> is the "something" can be anything and the "bad" can be hard to figure out.
> The more stuff you have in your DB, the harder it is to figure out.
>
> First, I'd make sure this is happening. Your logs should make it obvious.
> You might see some FATAL errors just before it starts over.  You MIGHT be
> able to nar

Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

Thanks, Blake.

Turns out we did add quite a few records recently, so maybe there was
something in there that it did not like all that much.

How can you tell which record it is choking on?  Is that your "1x1"
suggestion?  Or does the DEBUG option make that more clear?  I have my
indexing set to:

AppConfig[:indexer_records_per_thread]  = 25
AppConfig[:indexer_thread_count]= 2

for both PUI and Staff records. I believe you are suggesting it would most
easily be found using 1 and 1?  I can see where that could take a long
time. But it if is going to choke over and over on the same record, then
that may be the best way to address it.

Do you think if I just did staff indexing without PUI, that it would be
identified faster?  Or could it pass the staff side but then die on PUI
later?

I hope to try some of these ideas after hours today, so if you can confirm
that I've got the right idea, that would help.

Tom


On Tue, May 10, 2022 at 2:17 PM Blake Carver 
wrote:

> > Is this possible?
>
> Short answer, Yes, it's possible your indexer is starting over.
>
> Long answer. This can be tricky to figure out. Something is wrong, the
> indexer never wants to do that. Sometimes "something" "bad" gets into
> ArchivesSpace and the indexer will just crash and start over. The problem
> is the "something" can be anything and the "bad" can be hard to figure out.
> The more stuff you have in your DB, the harder it is to figure out.
>
> First, I'd make sure this is happening. Your logs should make it obvious.
> You might see some FATAL errors just before it starts over.  You MIGHT be
> able to narrow it down from that. That is, what group of records had that
> error in the logs? Maybe that narrows it down enough. You just got lucky!
>
> I don't think I've ever been so lucky. What I'd do next is set your
> loglevel to DEBUG and restart. If you're feeling lucky or just impatient or
> both, leave the indexer speed as is. You'll get more details out of the
> logs and you should be able to narrow it down better. Ideally, you want to
> run the indexers at 1x1, which means it could take forrreeevver to get
> back around to the crash again. If you're lucky, it'll crash on a record,
> you'll go look at that record, the problem will be obvious, and there will
> be much rejoicing. With it running 1x1 you should see exactly what's
> causing the fail. If it's not crashing on the same record every time
> ugh. That's an even longer answer.
>
>
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Tuesday, May 10, 2022 10:23 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] (re)indexing in 2.8.1
>
> I don't look at the logs a lot unless there are issues with ArchivesSpace,
> so maybe this is something normal. But, after a restart due to some
> complaints about database connectivity, it looks like our ArchivesSpace
> instance has decided to do a full reindex. The index log sure looks as if
> it is starting from scratch and running through the indexing of both PUI
> and Staff indexes.
>
> Is this possible?  Is it something that happens periodically and I just
> did not notice it? Nothing has changed in my data directory, so I don't see
> any reason for indexing to occur. Yet that is what the logs show.
>
> If it is doing this for some reason, and knowing that we restart
> periodically, it seems like we will get into a loop where indexing just
> keeps happening all the time. Also, it would be helpful to understand what
> caused this to happen.
>
> Any thoughts or experiences from those who have run this for longer would
> be appreciated. I'd like to understand if it would be a good idea to clear
> the data directory and perform a full index over the weekend rather than an
> unexpected and possibly never ending round in the background.
>
> Thanks,
> Tom
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] (re)indexing in 2.8.1

[Tom Hanstra]

I don't look at the logs a lot unless there are issues with ArchivesSpace,
so maybe this is something normal. But, after a restart due to some
complaints about database connectivity, it looks like our ArchivesSpace
instance has decided to do a full reindex. The index log sure looks as if
it is starting from scratch and running through the indexing of both PUI
and Staff indexes.

Is this possible?  Is it something that happens periodically and I just did
not notice it? Nothing has changed in my data directory, so I don't see any
reason for indexing to occur. Yet that is what the logs show.

If it is doing this for some reason, and knowing that we restart
periodically, it seems like we will get into a loop where indexing just
keeps happening all the time. Also, it would be helpful to understand what
caused this to happen.

Any thoughts or experiences from those who have run this for longer would
be appreciated. I'd like to understand if it would be a good idea to clear
the data directory and perform a full index over the weekend rather than an
unexpected and possibly never ending round in the background.

Thanks,
Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] gemfile issues - 2.8.1 and plugins

[Tom Hanstra]

Ok, so I missed the script step on the plugin. Did not read the manual
quite well enough. Once the additional bundling was done, the plugin seems
to be working.

Tom

On Fri, Apr 29, 2022 at 11:13 AM Tom Hanstra  wrote:

> I'm attempting to add the bulk_updater plugin to our 2.8.1 instance. A
> prerequisite for that is the digitization_work_order plugin.
>
> Installation of this worked ok in our test version of 3.2.0. But when
> installing on 2.8.1, I'm getting the error:
>
> WARNING: ERROR: initialization failed
> org.jruby.rack.RackInitializationException: Could not find gem 'write_xlsx
> (= 1.01.0) java' in any of the gem sources listed in your Gemfile.
>
> How do I go about  getting this fixed?  Do I have to use an older version
> of the plugins?  Or add something more to ArchivesSpace?
>
> Thanks,
> Tom
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
>

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] gemfile issues - 2.8.1 and plugins

[Tom Hanstra]

I'm attempting to add the bulk_updater plugin to our 2.8.1 instance. A
prerequisite for that is the digitization_work_order plugin.

Installation of this worked ok in our test version of 3.2.0. But when
installing on 2.8.1, I'm getting the error:

WARNING: ERROR: initialization failed
org.jruby.rack.RackInitializationException: Could not find gem 'write_xlsx
(= 1.01.0) java' in any of the gem sources listed in your Gemfile.

How do I go about  getting this fixed?  Do I have to use an older version
of the plugins?  Or add something more to ArchivesSpace?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Larger records and OAI - 3.2.0

[Tom Hanstra]

We are still having difficulties with larger records in OAI
searches/responses while using AS 3.2.0. I've just completed running
through the soft indexing process to be sure Solr (8.11.1) and AS are in
sync. But I'm still running into problems.

When I do these searches, I'm not getting any error messages in any of the
AS logs nor in the Solr logs. Small records come back reasonably quickly,
but larger records do not. They basically time out after a long period of
time.

Watching the server, the load average ticks up during the searches overall,
but most of the work is being done by AS. Solr does not seem to be using
much CPU during this time. I also tried increasing the database server CPU
to see if that would help, but it doesn't seem to be helping.

I'd like to better understand what has to be done when attempting an OAI
request. What services get hit the hardest?  Where might I have a
bottleneck that I've not yet addressed? The fact that smaller records work
but larger records do not indicates to me that the issue is not with Solr,
but the lack of any errors in the logs seems to indicate that ArchivesSpace
is doing what it is supposed to. So where might things be falling apart?

Thanks,
Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Solr review

[Tom Hanstra]

Thanks, Brian.

Some more questions, assuming that I've gotten the configuration and cores
set up correctly.

- Memory: In my Archivesspace 2.8.1 instance, Solr and ArchivesSpace are
sharing the same Java memory (as far as I can tell). When we split them up,
are there suggested amounts of memory for each?  Do I need more memory
given to Java in order to address having these separated out?

I'm trying a soft reindex to see if my Solr and ArchivesSpace are not in
sync, but one thing I notice is that memory for Solr itself, because it is
external, is much lower than my production server. What do others use
memory and thread wise?

It seems to me that I'm resource bound somewhere, but I've been unable to
track down where so far. It seems like soft indexing should be pretty
speedy, since Solr records are already in place, but it is going quite
slowly so far.

- Other configuration settings
Might there be other Solr configuration settings which need to be tweaked
to speed up processing? There are a lot more values displayed in 8.11.1
that seem like they could be changed.

Tom



On Mon, Apr 11, 2022 at 3:18 PM Brian Hoffman 
wrote:

> Hi Tom,
>
>
>
> I share your confusion – I personally do not bother with configsets when I
> set up my own solr instances. I think it is an either / or situation – you
> can use them if you want, or just create the core you need directly if you
> have installed solr natively and have the standard tools:
>
>
>
> % solr create -c archivesspace -d /path/to/archivesspace/solr
>
>
>
> Solr needs to be running, and after that command succeeds you should be
> able to verify through the admin app that the core exists and the schema
> has archivesspace field definitions.
>
>
>
> Let me know if any of this doesn’t make sense.
>
>
>
> Brian
>
>
>
> *From: *archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Date: *Monday, April 11, 2022 at 2:52 PM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *[Archivesspace_Users_Group] Solr review
>
> I'm reviewing the Solr documentation from ArchivesSpace and what I have
> learned from other Solr configurations locally. I'm wondering if I'm
> missing something.
>
> The ArchivesSpace documentation here:
>
> https://archivesspace.github.io/tech-docs/provisioning/solr.html
>
> suggests the need to copy the various Solr config files over to a
> configsets directory. In my case, with solr installed in /opt/solr that
> directory would be:
>
> /opt/solr/server/solr/configsets/archivessspace/conf
>
> and then copy the configuration files over to that location. But then I
> don't see anything that specifically tells Solr that that is where to find
> those files.
>
>
>
> I believe that it is simply the fact that the files exists down under the
> configured $SOLR_HOME directory that allows Solr to find these, but I'm
> trying to make sure. I have other Solr instances in which I've just got a
> "conf" directory under SOLR_HOME which is not under a configsets directory
> which seems sufficient to get Solr set up, but I'm trying to be sure. I"m
> still trying to figure out a problem with either AS or Solr in my 3.2.0
> instance, so I'm checking everything along the way. The core looks right
> (similar number of records after indexing), but maybe there is a mismatch
> somewhere.
>
>
>
> Thanks for any thoughts on this,
>
> Tom
>
> --
>
> *Tom Hanstra*
>
> *Sr. Systems Administrator*
>
> hans...@nd.edu
>
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Solr review

[Tom Hanstra]

I'm reviewing the Solr documentation from ArchivesSpace and what I have
learned from other Solr configurations locally. I'm wondering if I'm
missing something.

The ArchivesSpace documentation here:

https://archivesspace.github.io/tech-docs/provisioning/solr.html

suggests the need to copy the various Solr config files over to a
configsets directory. In my case, with solr installed in /opt/solr that
directory would be:

/opt/solr/server/solr/configsets/archivessspace/conf

and then copy the configuration files over to that location. But then I
don't see anything that specifically tells Solr that that is where to find
those files.

I believe that it is simply the fact that the files exists down under the
configured $SOLR_HOME directory that allows Solr to find these, but I'm
trying to make sure. I have other Solr instances in which I've just got a
"conf" directory under SOLR_HOME which is not under a configsets directory
which seems sufficient to get Solr set up, but I'm trying to be sure. I"m
still trying to figure out a problem with either AS or Solr in my 3.2.0
instance, so I'm checking everything along the way. The core looks right
(similar number of records after indexing), but maybe there is a mismatch
somewhere.

Thanks for any thoughts on this,
Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] OAI Harvesting issues with 3.2.0

[Tom Hanstra]

Thanks, Andrew,

So far, I've not been able to capture anything in the logs which shows
errors, though I'm still trying. I've bumped up the logging to try to give
a better idea of what is happening.

One thing I have seen is that the server load really jumps when we are
trying to do the OAI harvest. Something is taking a lot more resources
though the only process on the server which I can really see is the "java"
process running ArchivesSpace.

I anticipate having to come back to this and to the group after we have
some more data. If anyone is running 3.2.0 (with Java 11) and has ideas for
me, that would be great.

Tom

On Wed, Mar 30, 2022 at 5:33 AM Andrew Morrison <
andrew.morri...@bodleian.ox.ac.uk> wrote:

> If you post the error messages in your log files from around the time when
> you get an "Internal Server Error" it would help diagnose the problem. But
> here are some observations that might be relevant.
>
> Exporting EAD, whether from the staff interface or via the OAI-PMH
> service, uses both MySQL and Solr. The former to retrieve the resource and
> the IDs of its archival objects. The latter to retrieve the archival
> objects, although it checks whether the version in Solr is the same as in
> MySQL, and fetches from the database if not. So your problems could be with
> either, or both. Also, if Solr and MySQL are out-of-sync on your 3.2
> system, but in-sync on the 2.8.1 one, that could explain some of the
> difference in response time. You could try a soft re-index and see if that
> has any effect:
>
> https://archivesspace.github.io/tech-docs/administration/indexes.html
>
> Wherever it gets the records from, they're retrieved in batches of 20 at a
> time. Those are then converted from JSON to EAD. That is the CPU-intensive
> part, and single-threaded, so typically takes up most of the overall
> runtime. But if there's something about your infrastructure which makes
> retrieval slow, you could reduce total waiting time by increasing the batch
> size, which is possible by putting the following in
> *backend/plugin_init.rb* in a local plugin and restarting ArchivesSpace:
>
> module ASpaceExport
>   module LazyChildEnumerations
> PREFETCH_SIZE = 50
>   end
> end
>
> Your mileage may vary. Bigger batches will increase memory usage. And it
> might make a big difference for some collections, but none at all in
> others, because the exporter only ever requests siblings. Therefore a
> collection with a deeply-nested structure can require hundreds more batches
> than one which is shallow, despite having the same total number of archival
> objects in both, regardless of how high you set the prefetch size.
>
> The OAI-PMH service has an additional issue that it cannot stream its
> output. See here:
>
> https://archivesspace.atlassian.net/browse/ANW-1270
>
> Andrew.
>
>
> On 29/03/2022 17:37, Andy Boze wrote:
>
> Just to elaborate a bit on what Tom wrote, we are harvesting EAD records.
> I've done a bit of comparison, making OAI requests for the same records on
> 2.8.1 and 3.2. A record on 2.8.1 that took about 10 seconds , took about 3
> minutes on 3.2. A record that took about 3 minutes to respond on 2.8.1
> timed out on 3.2 after 20 minutes with an "Internal Server Error" message.
>
> Andy
>
> On 3/29/2022 11:57 AM, Tom Hanstra wrote:
>
> We have set up a test server running ArchivesSpace 3.2.0. As required,
> that means a separate Solr instance which I've installed on the same
> server.
>
> Most things have gone OK, but we are seeing some timeout issues with OAI
> harvesting tests. The harvest will address a few of the records but
> regularly receives "Internal Server Error" messages. What seems to be
> happening is that we are hitting certain records which time out. We've
> tried skipping over such records to see if it was just a bad record, but
> that will simply cause a failure a bit further down the line. Our time out
> is set for 20 minutes, which should be plenty of time. So these timeouts
> don't make much sense.
>
> These records are harvesting without similar issues on our 2.8.1 instance,
> so I would not expect this to be a record issue directly. Could it be
> something about how we have set up Solr? I see no errors in any of our
> ArchivesSpace or Solr logs, so I'm not sure how to debug this. Any
> suggestions?
>
> Thanks,
> Tom
>
> --
> *Tom Hanstra*
> /Sr. Systems Administrator/
> hans...@nd.edu <mailto:hans...@nd.edu> 
>
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mail

[Archivesspace_Users_Group] OAI Harvesting issues with 3.2.0

[Tom Hanstra]

We have set up a test server running ArchivesSpace 3.2.0. As required, that
means a separate Solr instance which I've installed on the same server.

Most things have gone OK, but we are seeing some timeout issues with OAI
harvesting tests. The harvest will address a few of the records but
regularly receives "Internal Server Error" messages. What seems to be
happening is that we are hitting certain records which time out. We've
tried skipping over such records to see if it was just a bad record, but
that will simply cause a failure a bit further down the line. Our time out
is set for 20 minutes, which should be plenty of time. So these timeouts
don't make much sense.

These records are harvesting without similar issues on our 2.8.1 instance,
so I would not expect this to be a record issue directly. Could it be
something about how we have set up Solr? I see no errors in any of our
ArchivesSpace or Solr logs, so I'm not sure how to debug this. Any
suggestions?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] 'Tis The Season For Ye Ol' "HOUR_OF_DAY: 2 -> 3" Error :-(

[Tom Hanstra]

Thanks, Blake. We actually got hit with this yesterday and I did not see
your message. A quick Google search found the issue, however.

What do other sites do to mitigate this?  Does everyone just wait until
after the DST change to see if things broke?  Or are they ways to
proactively mitigate this?

Also, can I expect the same thing when we "Fall back"?  Or is this somehow
just a Spring issue?

Thanks,
Tom

On Sun, Mar 13, 2022 at 2:34 PM Blake Carver 
wrote:

> Yet another reason to hate Daylight Savings Time...
> Some of you may notice troubles this morning. There's a weird little bug
> that happens when the clocks change. It doesn't hit ALL sites, but we had
> some hosted sites hit this year for the first time, so I thought I'd put
> this at the top O' the list.
>
> There's a JIRA here:
> https://archivesspace.atlassian.net/browse/ANW-1229
>
> It's PROBABLY the search_indexer user, and can be fixed by doing:
> UPDATE user set user_mtime = NOW(), system_mtime=NOW() where
> username='search_indexer';
>
> You might need to restart after updating the user
>
> If you're not quite so lucky, it could be hiding elsewhere, another user,
> or an AO or a Date.
>
> -- AO
> update archival_object set create_time = '2022-03-13 06:00:00',
> system_mtime = '2022-03-13 06:00:00'
> where create_time >= '2022-03-13 02:00:00' and create_time <= '2022-03-13
> 03:00:00';
> -- DATE
> update `date` set create_time = '2022-03-13 06:00:00', system_mtime =
> '2022-03-13 06:00:00'
> where create_time >= '2022-03-13 02:00:00' and create_time <= '2022-03-13
> 03:00:00';
> -- USER
> UPDATE user set user_mtime = NOW(), system_mtime=NOW() where
> username='SOME_OTHER_USER';
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Solr 2.8.1 errors - Solr not working

[Tom Hanstra]

Sorry. My problem. I updated solr configuration for my 3.2.0 testing and it
got into my 2.8.1 config file. So I was trying to point to the wrong Solr!

I found the error and fixed.

Tom

On Wed, Mar 9, 2022 at 9:46 AM Tom Hanstra  wrote:

> Our production instance of Archivesspace is not currently working. Solr
> seems to be having issues, but this is the first time Ive encountered this
> issue.
>
> I get the following error:
>
> E, [2022-03-09T09:31:33.671571 #14633] ERROR -- :
> [1ecbb955-3c05-4548-aac4-f3cb33906be0] 500: {"error":"Solr search f
> ailed: \n\n content=\"text/html; charset=ISO-8859-1\"/>\nError 40
> 4 Not Found\n\nHTTP ERROR 404\nProblem
> accessing /solr/archivesspace/select. Reason:
> \nNot FoundPowered by
> Jetty://
>  \n\n
>   \n\n
>  \
> n\n
>\n
>\n
>\n
> \n
>  \n
>  \n
>  \n
>   \n
>\n
>\n
>  \n
> \n\n
>  \n\n\n\n"}
>
>
> I'm not sure what is it trying to access or what might have gotten lost.
> Can anyone tell me what to look for and why we might be having this issue.
>
> Thanks,
> Tom
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
>

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Solr 2.8.1 errors - Solr not working

[Tom Hanstra]

Our production instance of Archivesspace is not currently working. Solr
seems to be having issues, but this is the first time Ive encountered this
issue.

I get the following error:

E, [2022-03-09T09:31:33.671571 #14633] ERROR -- :
[1ecbb955-3c05-4548-aac4-f3cb33906be0] 500: {"error":"Solr search f
ailed: \n\n\nError 40
4 Not Found\n\nHTTP ERROR 404\nProblem
accessing /solr/archivesspace/select. Reason:
\nNot FoundPowered by
Jetty://
 \n\n
  \n\n
 \
n\n
 \n
   \n
   \n
\n
   \n
 \n
 \n
  \n
 \n
   \n
 \n
\n\n
 \n\n\n\n"}


I'm not sure what is it trying to access or what might have gotten lost.
Can anyone tell me what to look for and why we might be having this issue.

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Indexing issues with 3.2.0

[Tom Hanstra]

We have a fairly large collection (765 K records)  which I'm indexing on a
3.2.0 instance. What I'm seeing now on both times it has completed is that
the indexer will complete (last message) and then go into the long gap
where it (I assume) is doing some post-indexing cleanup prior to reporting
a large number of deleted records. During that time, it reports an error in
connecting to the database.

I really doubt that the database is inaccessible during that time period.
Is this error just a matter of the server temporarily losing connection,
which is reestablished later?  Or is this something that could be causing
actual issues for the indexing?  What is actually going on during that gap
between the time when it says the last of the records is indexed and the
time it reports records being deleted? For us, that gap can last several
hours.

Thanks in advance for any help you can provide to clarify what happens.

Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Java 11 and 3.2.0?

[Tom Hanstra]

The documentation for ArchviesSpace 3.2.0 indicates that Java 11 is
supported but most information I see is for Java 1.8 (which I assume is
"Java 8"). Are we OK to use Java 11 for 3.2.0 or should I stick with 1.8?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] log4j vulnerability in ArchivesSpace?

[Tom Hanstra]

Right, it is bad. I'm digging around at everything this morning looking for
places that might be vulnerable.

There are a couple of gems in the gems directory which use older versions
of log4j (ladle-0.2.0-java, mizuno-0.6.11). No idea where those come into
play with the overall software.

Tom

On Sat, Dec 11, 2021 at 8:46 AM Blake Carver 
wrote:

> Almost certainly not, there's no absolutes in this stuff, but from
> everything I've read it's currently not vulnerable.
>
> This is a bad vulnerability, log4j is all over the place.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Saturday, December 11, 2021 8:21 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] log4j vulnerability in
> ArchivesSpace?
>
> There is a lot of buzz right now about the log4j exploit being used
> against Java applications. Does anyone know if ArchivesSpace is vulnerable
> to these exploits?
>
> Tom
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] log4j vulnerability in ArchivesSpace?

[Tom Hanstra]

There is a lot of buzz right now about the log4j exploit being used against
Java applications. Does anyone know if ArchivesSpace is vulnerable to these
exploits?

Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Issues with spreadsheet load

[Tom Hanstra]

Thanks, all, for the confirmation that this is happening so regularly
elsewhere. That helps.

Valerie: I think your suggestion of loading CSV seems like a good one. In
my testing, the same data which had problems as an xslx file worked fine as
csv. I've suggested that to our group here and will follow up with them to
see if that helps get things working better.

Hopefully this will be able to be fixed in upcoming versions.

Tom

On Tue, Nov 30, 2021 at 12:45 PM Christine Di Bella <
christine.dibe...@lyrasis.org> wrote:

> Hi everyone,
>
>
>
> In addition to this discussion, please feel free to add comments to the
> existing JIRA for this issue. It’s a hard one to search for because of the
> usual challenges of natural language, but it’s at
> https://archivesspace.atlassian.net/browse/ANW-1427.
>
>
>
> Christine
>
>
>
> Christine Di Bella
>
> ArchivesSpace Program Manager
>
> christine.dibe...@lyrasis.org
>
> 800.999.8558 x2905
>
> 678-235-2905
>
>
>
>
>
> [image: ASpaceOrgHomeMedium]
>
>
>
>
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of 
> *Valerie
> Addonizio
> *Sent:* Tuesday, November 30, 2021 11:44 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Issues with spreadsheet load
>
>
>
> I have also encountered this issue, and want to suggest, purely as a means
> of troubleshooting, to try and upload the same data using the CSV version
> of the spreadsheet
> <https://github.com/archivesspace/archivesspace/tree/master/templates>,
> and specifically Save As-ing as an UTF-8 CSV using Excel. We were never
> able to definitively determine if this made a difference, but it’s
> something I wanted to share as an idea on the path to troubleshooting. If
> you choose to try the CSV route, there are some quirks. One is that any
> Publish value that you have set to TRUE or FALSE in the xlsx version needs
> to be submitted as a 1 or 0 respectively in the CSV version.
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Noah
> Huffman
> *Sent:* Tuesday, November 30, 2021 11:24 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Issues with spreadsheet load
>
>
>
> Hi Tom,
>
>
>
> We have been experiencing this same issue for a while on our production
> v2.8.1 instance and on a test version of v3.1.0.
>
>
>
> If we attempt to load a spreadsheet that does not validate it not only
> fails, but essentially wrecks the importer for all future imports, even
> spreadsheets that are properly formatted. Any subsequent import attempt
> will return a server error in the UI. I’m also seeing the same error you
> reported in my log file.
>
>
>
> The only fix we have found is to restart the application.
>
>
>
> I thought I had submitted a bug report for this, but I’ll double-check now.
>
>
>
> -Noah
>
>
>
> 
>
> Noah Huffman
>
> Archivist for Metadata, Systems, and Digital Records
>
> David M. Rubenstein Rare Book & Manuscript Library
>
> Duke University | 919-660-5982
>
> http://library.duke.edu/rubenstein/
>
> Pronouns: he / him / his
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Tom
> Hanstra
> *Sent:* Tuesday, November 30, 2021 11:08 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] Issues with spreadsheet load
>
>
>
> I'm trying to track down some issues we are having with the load of a
> spreadsheet. I'm wondering if I have some database corruption behind the
> scenes or just have some bad spreadsheet info.
>
>
>
> We are running ArchivesSpace v 2.8.1. When trying to load the spreadsheet,
> we consistently get the error in the background log of:
>
>
>
> E, [2021-11-30T10:59:08.769519 #7965] ERROR -- : Thread-3238: Unhandled
> exception!
> E, [2021-11-30T10:59:08.770805 #7965] ERROR -- :
> invalid byte sequence in UTF-8
>
> I got this error just with the Only Validate option chosen, but I don't
> know enough about what Only validate does.
>
>
> We've also now gotten this with a couple of different spreadsheets against
> a few different records. We have not found a spreadsheet that wil

[Archivesspace_Users_Group] Issues with spreadsheet load

[Tom Hanstra]

I'm trying to track down some issues we are having with the load of a
spreadsheet. I'm wondering if I have some database corruption behind the
scenes or just have some bad spreadsheet info.

We are running ArchivesSpace v 2.8.1. When trying to load the spreadsheet,
we consistently get the error in the background log of:

E, [2021-11-30T10:59:08.769519 #7965] ERROR -- : Thread-3238: Unhandled
exception!
E, [2021-11-30T10:59:08.770805 #7965] ERROR -- :
invalid byte sequence in UTF-8

I got this error just with the Only Validate option chosen, but I don't
know enough about what Only validate does.

We've also now gotten this with a couple of different spreadsheets against
a few different records. We have not found a spreadsheet that will work as
things currently stand.

How can I best track down where the issue lies. Might I have an issue with
the database? Or just bad data somewhere in this spreadsheet? Is there a
way to validate an archivesspace database?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] API output - extra unicode

[Tom Hanstra]

Brian (and others),

The data in the database should be UTF-8 as far as I can tell. So, I think
this has to be happening at the API export level. Is there anything
specific that needs to be done to have the API know that this is UTF-8 data?

Tom

On Fri, Sep 3, 2021 at 11:42 AM Brian Harrington <
brian.harring...@lyrasis.org> wrote:

> Hi Tom,
>
>
>
> In my experience \u00c3 appearing in anything is almost always a sign of
> encoding issues.  I would make sure that everything is UTF-8 all the way
> through.
>
>
>
> Brian
>
>
>
> *From: * on
> behalf of Tom Hanstra 
> *Reply-To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Date: *Friday, September 3, 2021 at 11:06 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *[Archivesspace_Users_Group] API output - extra unicode
>
>
>
> On our local version of ArchivesSpace, we are testing API output and are
> finding that we are getting extra Unicode characters on export. It looks
> like the data is right in the database, but doesn't quite come out right
> from the API extract. It looks like there is an extra unicode character
> added (in some of the code we reviewed, this was either \u00c3 or \u00a2).
>
>
>
> Where might we have something set incorrectly?  Where might the extra data
> be coming from or have been introduced along the way?
>
>
>
> Thanks,
>
> Tom
>
>
>
> --
>
> *Tom Hanstra*
>
> *Sr. Systems Administrator*
>
> hans...@nd.edu
>
>
>
> [image: Image removed by sender.]
> _______
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] API output - extra unicode

[Tom Hanstra]

On our local version of ArchivesSpace, we are testing API output and are
finding that we are getting extra Unicode characters on export. It looks
like the data is right in the database, but doesn't quite come out right
from the API extract. It looks like there is an extra unicode character
added (in some of the code we reviewed, this was either \u00c3 or \u00a2).

Where might we have something set incorrectly?  Where might the extra data
be coming from or have been introduced along the way?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] MariaDB v. MySQL

[Tom Hanstra]

Either will work (or anything that is MySQL compatible. We use AWS Aurora).
The main point to me of that statement is to be sure not to run production
level ArchivesSpace using the local, base level database which is for
testing purposes only.

So use either one. For the ArchivesSpace database, there should be no
difference in functionality.

Tom

On Wed, Aug 18, 2021 at 9:21 AM Kat Spears  wrote:

> Our IT Manager had a question as we are preparing to implement
> ArchivesSpace. Perhaps someone could answer her question (below)? Any
> feedback is appreciated.
>
>
>
> Thanks,
>
> kat
>
>
>
> Her message to me was:
>
> Under the “implementing ArchivesSpace” page it says
>
>
>
> *“The embedded database is for testing purposes only. You should use MySQL
> or MariaDB for any data intended for production, including data in a test
> instance that you intend to move over to a production instance.”*
>
>
>
> Question is – how did they address that need?
>
>
>
> Are people using MySQL or MariaDB, which is better to tie into
> ArchivesSpace?
>
>
>
>
>
> *__*
>
>
>
>
> *Katarina Spears **Library and Archives Manager*
>
> *O *804.262.9887 ext. 342
>
> k...@lewisginter.org 
>
>
>
> 1800 Lakeside Avenue
> Richmond, VA 23228
>
>
>
> [image: /Volumes/Clients A-L/Lewis-Ginter/16 Projects/16LGB1710 Lewis
> Ginter Brand Refresh/06_Final/Wordmark/PRIMARY_FullMark/COLOR/RGB
> (digital)/PNG/LGBG_LogoPrimary_RGB_forEmailSignature.png]
> <http://www.lewisginter.org/>
>
>
> _______
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Large record update missing PUI data

[Tom Hanstra]

In our testing of a local ArchivesSpace instance, we updated a very large
record with some additional information. Information was added as expected
on the Staff side. But on the Public interface, the items show up as listed
in the collection organization, but when a user attempts to access the
records, the page says they do not exist.

Did we miss an update step somewhere or did the record just not get updated
properly on the PUI side?  Is there a way to reindex just this one record
again or do I have to fully update the PUI index to get this addressed. I'm
looking for the best way to handle this with the least amount of
interruptions.

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Requests and postfix

[Tom Hanstra]

I'm running ArchivesSpace on an AWS server and the preferred/configured
email service is postfix. It looks as if that is either not a supported
service or I have not quite figured out how to set it up within the
configuration file.

If I'm not using sendmail, what settings should I use in the configuration
file?  Or is sendmail a requirement for running requests? I don't find any
info in the documentation about other options besides sendmail.

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] config.rb options

[Tom Hanstra]

As we work with ArchivesSpace here, we are figuring out that the confg.rb
file has components which could be updated by various groups. Is there a
way to break up the file so that each group can maintain their portion but
all of it is combined to properly configure ArchivesSpace? How might others
handle this?

Thanks
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Errors in ArchivesSpace logs

[Tom Hanstra]

I've been reviewing my ArchivesSpace logs and see a variety of ERROR
messages in the logs, particularly in the backend log, which I've broken
out separately, but also in the indexing logs.  These seem to be a variety
of Ruby and/or Jetty errors.

How do I determine if any of these rise to the level of issues to be
investigated?  Are these to be expected or are they indicative of
problems?  And if they are problems, where do I look to figure out what
is wrong?  In most cases, we have not seen ArchivesSpace run into issues,
though we did have to restart on 5/27 around 11:45, which may be related to
the 5/27 errors below.

Here are a few of the errors I've seen (minus a lot of detail):

E, [2021-05-18T12:29:47.624349 #2342] ERROR -- : Thread-3412: Unhandled
exception!
E, [2021-05-18T12:29:47.635490 #2342] ERROR -- :
invalid byte sequence in UTF-8
org/jruby/ext/strscan/RubyStringScanner.java:261:in `scan'


E, [2021-05-20T09:43:21.938983 #2326] ERROR -- : Thread-2852: UNEXPECTED
EXCEPTION on bulkimport load! undefined method
`title' for nil:NilClass
E, [2021-05-20T09:43:21.961757 #2326] ERROR -- : Thread-2852:
["/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8089-back
end.war-_-any-/webapp/WEB-INF/app/lib/bulk_import/import_archival_objects.rb:114:in
`process_row'",

E, [2021-05-21T10:20:53.012297 #6626] ERROR -- : Thread-7622: Unhandled
exception!
E, [2021-05-21T10:20:53.471462 #6626] ERROR -- :
uninitialized constant Kernel::RepositoryWithAgent
Did you mean?  Repository
org/jruby/RubyModule.java:3760:in `const_missing'
org/jruby/RubyModule.java:3707:in `const_get'

E, [2021-05-27T09:14:00.663543 #6626] ERROR -- : Thread-8388: Unhandled
exception!
E, [2021-05-27T09:14:00.686600 #6626] ERROR -- :
key not found: 1484929
org/jruby/RubyHash.java:1261:in `fetch'

and from the index logs...

E, [2021-05-27T09:14:00.743424 #6626] ERROR -- : Thread-2930:
/home/app/archivesspace/data/tmp/jetty-0.0.0.0-8091-indexe
r.war-_aspace-indexer-any-/webapp/WEB-INF/app/lib/large_tree_doc_indexer.rb:103:in
`block in index_paths_to_root'
org/jruby/RubyArray.java:1851:in `each_slice'


Any help in figuring things out would be useful.

Thanks,
Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] lcnaf plugin

[Tom Hanstra]

Thanks. That helps a lot. I figured history factored in as it always does
with these kinds of things. But coming in cold, it is especially confusing
now that I know there are two versions. I'll assume that the included
version is sufficient until someone here says differently.

It would help if this was documented a bit more. A simple "README" file in
the lcnaf GitHub repository mentioning that it is an even more enhanced
version of the included version and explaining the differences would go a
long way in explaining the situation. I'm not sure who maintains that code,
but it certainly would have helped me.

And thanks for pointing me to the wiki. It looks like I'm going to have to
do some perusing of that in addition to the tech docs in order to really
understand how to support this software.

Tom


On Thu, May 20, 2021 at 3:13 PM Christine Di Bella <
christine.dibe...@lyrasis.org> wrote:

> Hello Tom,
>
>
>
> The LCNAF plugin is a very rare case in that it’s both bundled with the
> standard ArchivesSpace application and, because of its popularity, lives on
> its own in case it needs to be updated between our releases due to changes
> in LC’s service. The program team has discussed doing only one or the other
> because it is undeniably duplicative for code maintainers and confusing for
> users (though, on the users side, most people use the one that comes
> bundled with ArchivesSpace and don’t realize there is a separate one). The
> functionality of both LCNAF plugins is identical. For now we’ve continued
> to keep it the way it is pending some larger discussions.
>
>
>
> Virtually all other plugins only live on our their own, and most plugins
> are authored by and maintained by community members, not the ArchivesSpace
> program. The community-maintained Awesome ArchivesSpace list links to a lot
> of great ones: https://github.com/archivesspace/awesome-archivesspace.
>
>
>
> For some additional context, there’s some information about plugins and
> considerations for turning a plugin into core code at
> https://archivesspace.atlassian.net/wiki/spaces/ADC/pages/916062227/Making+a+Plug-In+part+of+the+Core+Code.
> That document also links to an outline of an evaluation process for
> considering feature contributions, including whether something might best
> be handled as a plugin:
> https://archivesspace.atlassian.net/wiki/spaces/ADC/pages/360546309/ArchivesSpace+Process+for+Evaluating+Potential+Feature+Contributions+to+the+Core+Code
> .
>
>
>
> I hope this helps. If you have other questions, please let us know.
>
>
>
> Christine
>
>
>
> Christine Di Bella
>
> ArchivesSpace Program Manager
>
> christine.dibe...@lyrasis.org
>
> 800.999.8558 x2905
>
> 678-235-2905
>
>
>
>
>
> [image: ASpaceOrgHomeMedium]
>
>
>
>
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Tom
> Hanstra
> *Sent:* Thursday, May 20, 2021 12:23 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] lcnaf plugin
>
>
>
> Ok, so I did not see that lcnaf was included in ArchivesSpace right from
> the beginning. I thought I had to get the files from GitHub as I would with
> other plugins, but I see that they are there. In fact several are there.
>
>
>
> So I need to revise my overall question to:
>
>
>
> - Why is some code added to the zip file but left as a plugin while other
> code is added to the software?
>
>
>
> In this instance, lcnaf is there but I need to update the config file to
> turn it on. With other plugins, I have to get the files from somewhere else
> and turn them on. And maybe do more than just put them in the directory
> (for instance, oauth needs to be compiled). How is it determined what goes
> where and how it is or is not included?
>
>
>
> Sorry, but this is rather confusing to the newbie.
>
>
>
> Thanks
>
> Tom
>
>
>
> On Thu, May 20, 2021 at 10:51 AM Joshua D. Shaw <
> joshua.d.s...@dartmouth.edu> wrote:
>
> Hi Tom
>
>
>
> To enable any plugin in the /plugins directory, you need to update the
> config file. Specifically, the entry
>
>
>
> AppConfig[:plugins] = []
>
>
>
> The array contents are just the directory names of the plugins that you
> want to enable
>
>
>
> So, to enable just the lcnaf plugin, you would want
>
>
>
> AppConfig[:plugins] = ['lcnaf']
>
>
>
> https://archivesspace.github.io/tech-docs/ is a good resource.
>
>
>
> jds
>
>
> --
>
> 

Re: [Archivesspace_Users_Group] lcnaf plugin

[Tom Hanstra]

Ok, so I did not see that lcnaf was included in ArchivesSpace right from
the beginning. I thought I had to get the files from GitHub as I would with
other plugins, but I see that they are there. In fact several are there.

So I need to revise my overall question to:

- Why is some code added to the zip file but left as a plugin while other
code is added to the software?

In this instance, lcnaf is there but I need to update the config file to
turn it on. With other plugins, I have to get the files from somewhere else
and turn them on. And maybe do more than just put them in the directory
(for instance, oauth needs to be compiled). How is it determined what goes
where and how it is or is not included?

Sorry, but this is rather confusing to the newbie.

Thanks
Tom

On Thu, May 20, 2021 at 10:51 AM Joshua D. Shaw 
wrote:

> Hi Tom
>
> To enable any plugin in the /plugins directory, you need to update the
> config file. Specifically, the entry
>
> AppConfig[:plugins] = []
>
> The array contents are just the directory names of the plugins that you
> want to enable
>
> So, to enable just the lcnaf plugin, you would want
>
> AppConfig[:plugins] = ['lcnaf']
>
> https://archivesspace.github.io/tech-docs/ is a good resource.
>
> jds
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Thursday, May 20, 2021 9:09 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] lcnaf plugin
>
> Hi, ArchivesSpace users,
>
> I continue my quest to get our site set up and have been told that I need
> to install the lcnaf plugin. I've got a few questions:
>
> First, where is the documentation on how to install the plugin? The code
> in github does not have so much as a README file to give me information on
> how to install it. Do I just grab the code and plunk it into the plugins
> directory? Or is there more to be done?
>
> Second, how does ArchivesSpace decide what plugins get pulled into code
> and which do not. This one, from what I can tell, has been around for a
> long time and yet continues to be a plugin that is added after the fact.
> Yet other plugins have been added into the code. Just curious how that gets
> decided.
>
> Thanks,
> Tom
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] lcnaf plugin

[Tom Hanstra]

Hi, ArchivesSpace users,

I continue my quest to get our site set up and have been told that I need
to install the lcnaf plugin. I've got a few questions:

First, where is the documentation on how to install the plugin? The code in
github does not have so much as a README file to give me information on how
to install it. Do I just grab the code and plunk it into the plugins
directory? Or is there more to be done?

Second, how does ArchivesSpace decide what plugins get pulled into code and
which do not. This one, from what I can tell, has been around for a long
time and yet continues to be a plugin that is added after the fact. Yet
other plugins have been added into the code. Just curious how that gets
decided.

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Indexing questions

[Tom Hanstra]

I'm trying to understand what a "soft reindex" accomplishes and when it
would be best used. Is it just a situation where we should always try that
first and, if it does not fix something, then decide to possibly do a full
reindex? Or are there certain situations where soft reindex is regularly
the thing to address the situation?

Also, does it gain anything to reindex by removing the "state" data and
having ArchivesSpace fully reindex but leave the Solr files in place?  It
would allow searches to take place while indexing is happening. Would
reindexing this way also update/fix Solr at the same time? Or could it
leave bad Solr data in place and thus cause other issues?

Thanks,
Tom



-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] API authorization

[Tom Hanstra]

Thanks, Valerie,

It turned out that we were not getting the proper headers passed into the
calls we were making. Once we got things called properly, we were able to
retrieve the desired information.

One more thing checked off the list...

Tom

On Thu, Apr 29, 2021 at 11:42 AM Valerie Addonizio 
wrote:

> Tom,
>
>
>
> The permissions for the API are the same as those set in the staff
> interface, so if your user is called api-user123, set the permissions and
> groups for api-user123 via the staff interface (in each repository where
> you expect that user to work) and those permissions will persist while
> using the API with that account. I hope this addresses your question!
>
>
>
> -Valerie
>
>
>
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> *On Behalf Of *Tom
> Hanstra
> *Sent:* Thursday, April 29, 2021 9:43 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] API authorization
>
>
>
> We are testing the API interface. So far, we have been able to get
> authentication to work via the API but are running into various
> authorization issues.  Some activity that we want to do is being denied.
>
>
>
> Where can I find documentation on API authorization/access?  I'm guessing
> we simply have not given the user access properly, but I have not been able
> to find where to review that information so far.
>
>
>
> Thanks,
>
> Tom
>
>
>
> --
>
> *Tom Hanstra*
>
> *Sr. Systems Administrator*
>
> hans...@nd.edu
>
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] API authorization

[Tom Hanstra]

We are testing the API interface. So far, we have been able to get
authentication to work via the API but are running into various
authorization issues.  Some activity that we want to do is being denied.

Where can I find documentation on API authorization/access?  I'm guessing
we simply have not given the user access properly, but I have not been able
to find where to review that information so far.

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Server slows down over time

[Tom Hanstra]

We continue our trek to get ArchivesSpace running locally. We are currently
doing a lot of testing with OAI and other background activity.

What we are seeing in our work is that, over time, the server will slow
down. Searches and updates which take just a few seconds after a restart
later get to a point where they take 30-40 seconds or longer. Then we
restart and things speed up again. I saw similar slowdowns when indexing
but did not have the ability to restart then because that would have
interrupted indexing.

Have others seen these types of issues?  Is this a Java thing that might be
happening? Or something else. My current Java settings are the values that
come out of the box:

ASPACE_JAVA_XMX="-Xmx1024m"
ASPACE_JAVA_XSS="-Xss2m"

I did increase these when I was indexing so perhaps they need to be tweaked
again?  Are there ways of figuring out the proper values or is it just
trial and error?

Or am I barking up the wrong tree and it is something else entirely.

Thanks for any suggestions you might have,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Indexed - but still some issues

[Tom Hanstra]

My understand/reading regarding this setting is that it is the thread size
for Java, and has to be big enough to handle the work but cannot be too big
either. The default of 2m seems very low. I have been working with the 1g
size lately but am not sure that is optimal either. I'd be curious to know
if there are optimal numbers that people have found useful.

Tom

On Wed, Mar 24, 2021 at 12:55 PM Trevor Thornton  wrote:

> I'm doing this exact thing right now and I wanted to confirm that the 'g'
> here shouldn't be an 'm':
> ASPACE_JAVA_XSS="-Xss1g"
>
> The default looks to be:
> ASPACE_JAVA_XSS="-Xss2m"
>
> https://github.com/archivesspace/archivesspace/blob/master/launcher/archivesspace.sh#L105
>
> On Wed, Mar 24, 2021 at 11:43 AM Tom Hanstra  wrote:
>
>> Hmmm. So I finally tracked the issue down to my Java settings. For some
>> reason, before/during indexing, Java settings of:
>>
>> ASPACE_JAVA_XMX="-Xmx8192m"
>> ASPACE_JAVA_XSS="-Xss4g"
>>
>> worked fine. But, once indexing was complete, those settings were somehow
>> unusable. Staff connection would not work, searches failed, etc.
>>
>> But when I crank down the thread size to:
>>
>>ASPACE_JAVA_XMX="-Xmx8192m"
>>ASPACE_JAVA_XSS="-Xss1g"
>>
>> everything goes back to working. There was a server reboot in there as
>> well as numerous restarts with config.rb changes and that was the only
>> change that I found could address the issues. Through a number of restarts,
>> there was nothing in the logs to indicate the issue. Finally got an error
>> that suggested Java space issues and tried the lower value and things now
>> work (as far as I've tested at least).
>>
>> Quite a journey so far...
>>
>> Tom
>>
>>
>> On Wed, Mar 24, 2021 at 9:54 AM Peter Heiner  wrote:
>>
>>> Tom Hanstra wrote on 2021-03-24 09:25:24:
>>> > So, I am now further than I've been yet, at least based upon the logs.
>>> But
>>> > I still have something not working properly, because when I do a
>>> search and
>>> > find output, nothing is coming up when I try to look at the found
>>> record. I
>>> > get the error:
>>> >
>>> > *We're sorry, but something went wrong.*
>>> >
>>> > When I've seen that in other applications, it has been a rails issue.
>>> Not
>>> > sure if that is what is happening here as well.
>>>
>>> This error should have a corresponding event with at least WARNING level
>>> in
>>> the frontend/public log (depending on which gave you the error) that you
>>> can
>>> correlate by event time.
>>>
>>> > Also, the "Found In" links for the records come back with:
>>> >
>>> > *The record you've tried to access does not exist or may have been
>>> > removed.The record you've tried to access does not exist or may have
>>> been
>>> > removed.*
>>>
>>> This would point to the fact that your database and index are not in
>>> sync.
>>> Given your struggles with indexing I'm starting to wonder if you're
>>> reading
>>> from the Solr instance you're writing to.
>>>
>>> p
>>>
>>> ___
>>> Archivesspace_Users_Group mailing list
>>> Archivesspace_Users_Group@lyralists.lyrasis.org
>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>>
>>
>>
>> --
>> *Tom Hanstra*
>> *Sr. Systems Administrator*
>> hans...@nd.edu
>>
>>
>> ___
>> Archivesspace_Users_Group mailing list
>> Archivesspace_Users_Group@lyralists.lyrasis.org
>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>
>
>
> --
> Trevor Thornton
> Applications Developer, Digital Library Initiatives
> North Carolina State University Libraries
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] More indexing woes?

[Tom Hanstra]

I have two instances which have successfully completed indexing. And with
the lowering of the Jave thread size, searches and other access is now
working.

But...

When I choose the "Repositories" button at the top of the Public interface,
I get an error message:

E, [2021-03-24T13:41:12.798448 #3141] ERROR -- :
[d9a19e62-859c-4ddb-b73f-7985974b1f4c] No repository records found!

and get nothing in terms of repositories.  Also, I'm not getting nearly
enough Collections. Our hosted version has 1539 collections. My server has
only 295.

>From the Staff side, I'm getting nothing when doing any "Browse" options.
Each of the pulldowns results in: "No records found". Searches in the Staff
interface do return values.

Do I possibly have some other configuration settings not set properly? Or
did my indexing not *really* work and I need to restart indexing yet again?

Again, my lack of experience may be an issue, but things still don't
seem to work.

Thanks,
Tom


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Indexed - but still some issues

[Tom Hanstra]

Hmmm. So I finally tracked the issue down to my Java settings. For some
reason, before/during indexing, Java settings of:

ASPACE_JAVA_XMX="-Xmx8192m"
ASPACE_JAVA_XSS="-Xss4g"

worked fine. But, once indexing was complete, those settings were somehow
unusable. Staff connection would not work, searches failed, etc.

But when I crank down the thread size to:

   ASPACE_JAVA_XMX="-Xmx8192m"
   ASPACE_JAVA_XSS="-Xss1g"

everything goes back to working. There was a server reboot in there as well
as numerous restarts with config.rb changes and that was the only change
that I found could address the issues. Through a number of restarts, there
was nothing in the logs to indicate the issue. Finally got an error that
suggested Java space issues and tried the lower value and things now work
(as far as I've tested at least).

Quite a journey so far...

Tom


On Wed, Mar 24, 2021 at 9:54 AM Peter Heiner  wrote:

> Tom Hanstra wrote on 2021-03-24 09:25:24:
> > So, I am now further than I've been yet, at least based upon the logs.
> But
> > I still have something not working properly, because when I do a search
> and
> > find output, nothing is coming up when I try to look at the found
> record. I
> > get the error:
> >
> > *We're sorry, but something went wrong.*
> >
> > When I've seen that in other applications, it has been a rails issue. Not
> > sure if that is what is happening here as well.
>
> This error should have a corresponding event with at least WARNING level in
> the frontend/public log (depending on which gave you the error) that you
> can
> correlate by event time.
>
> > Also, the "Found In" links for the records come back with:
> >
> > *The record you've tried to access does not exist or may have been
> > removed.The record you've tried to access does not exist or may have been
> > removed.*
>
> This would point to the fact that your database and index are not in sync.
> Given your struggles with indexing I'm starting to wonder if you're reading
> from the Solr instance you're writing to.
>
> p
>
> ___________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Indexed - but still some issues

[Tom Hanstra]

Whew! Ok, my logs now say that both Staff and PUI have been indexed. So I
got that far finally. Thanks to both Blake and Mark for suggestions. The
combination of staggering Staff and then PUI indexing as well as moving to
1 thread seems to have helped.

So, I am now further than I've been yet, at least based upon the logs. But
I still have something not working properly, because when I do a search and
find output, nothing is coming up when I try to look at the found record. I
get the error:

*We're sorry, but something went wrong.*

When I've seen that in other applications, it has been a rails issue. Not
sure if that is what is happening here as well.

Also, the "Found In" links for the records come back with:

*The record you've tried to access does not exist or may have been
removed.The record you've tried to access does not exist or may have been
removed.*

The logs are prolific, but I don't see anything right off to tell me what
is missing/broken. What should I be looking for?  Once I get indexing to
the point where they are both showing:

I, [2021-03-24T09:17:08.655860 #10795]  INFO -- : Thread-2868: Staff
Indexer [2021-03-24 09:17:08 -0400] Index round complete
I, [2021-03-24T09:17:18.314866 #10795]  INFO -- : Thread-3200: PUI Indexer
[2021-03-24 09:17:18 -0400] Index round complete

Is there more to be done behind the scenes? Do I need to restart
something?  Are there log entries that I should be looking for to show what
is going on?

Another possible red flag is that when I do searches, the results are
different than what I see on our hosted site. Not sure why that might be
either.

Thanks to those who have helped me get this far. Hopefully it is not going
to be overly difficult to get the rest of the way...

Tom



-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

 records, etc.)
>
>
> Mark
>
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Monday, March 22, 2021 11:21 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] PUI indexing issues
>
> Thanks, Blake.
>
> In your testing, how big was the repository that you were testing against.
> Mine has "763368 archival_object records" and I consistently get into the
> 670K range for staff and 575 range for PUI before things really slow down.
> I'm now trying to really increase the Java settings to see if that will
> help. So far, the problem is similar: real slow downs after zipping through
> the first records. I'll also try some of the settings you have there to see
> if fewer but larger threads work better than multiple smaller threads.
>
> Tom
>
> On Mon, Mar 22, 2021 at 10:52 AM Blake Carver 
> wrote:
>
> I did some experimenting this weekend, messing around with indexer speeds,
> and found I could get it to succeed with the right indexer settings. I
> think the answer is going to be "it depends" and you'll need to experiment
> with what works on your set up with your data. I started with the defaults,
> then dropped it to realy slow (1 thread 1 per), then just tried to dial
> it up and down. The last one I tried worked fine, it was fast enough to
> finish in a reasonable amount of time and didn't slow down or crash. Your
> settings may not look like this, but here's something to try.
>
> AppConfig[:pui_indexer_records_per_thread] = 50
> AppConfig[:pui_indexer_thread_count] = 1
>
>
> So some extra detail for the mailing list archives... if your site keeps
> crashing before the indexers finish and you're not seeing any particular
> errors in the logs that make you think you have a problem with your data,
> try turning the knobs on your indexer speed and see if that helps.
>
> It looks like maybe the indexer just eats up too much memory on BIG
> records and having too many (too many being 15ish) threads running causes
> it to crash. I know BIG is pretty subjective, if you have a bunch of
> resources (maybe a few thousand) AND those resources all have ALLOTA (maybe
> a few thousand) children with ALLOTA subjects/agents/notes/stuff, then you
> might hit this problem. Seems like it's not the total number of resources,
> it's probably because those resources are big/complex/deep.
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Thursday, March 18, 2021 11:24 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] PUI indexing issues
>
> Dave,
>
> Thanks for the suggestion, but unless there is some direct limitation
> within Solr, that should not be an issue. My disk is at only about 50% of
> capacity and Solr should be able to expand as needed. In my case, I don't
> think there has been much addition to Solr because I'm reindexing records
> which have been indexed already. So the deleted records are growing, but
> not the overall number of records. My index is currently at about 6GB.
>
> Any other thoughts out there?
>
> Thanks,
> Tom
>
> On Thu, Mar 18, 2021 at 10:51 AM Mayo, Dave  wrote:
>
> This is a little bit of a shot in the dark, but have you looked at disk
> space on whatever host Solr is resident on? (the ASpace server if you’re
> not running an external one)?
>
> A thing we’ve hit a couple times is that Solr, at least in some
> configurations, needs substantial headroom on disk to perform well – I
> think it’s related to how it builds and maintains the index?  So it might
> be worth looking to see if Solr is filling up the disk enough that it can’t
> efficiently handle itself.
>
>
>
> --
>
> Dave Mayo (he/him)
>
> Senior Digital Library Software Engineer
> Harvard University > HUIT > LTS
>
>
>
> *From: * on
> behalf of Tom Hanstra 
> *Reply-To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Date: *Wednesday, March 17, 2021 at 11:43 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] PUI indexing issues
>
>
>
>
>
>
>
> - What really bothers me is the slowdown. That indicates to me that some
> resource is being lost along the way. A

Re: [Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

Thanks, Blake.

In your testing, how big was the repository that you were testing against.
Mine has "763368 archival_object records" and I consistently get into the
670K range for staff and 575 range for PUI before things really slow down.
I'm now trying to really increase the Java settings to see if that will
help. So far, the problem is similar: real slow downs after zipping through
the first records. I'll also try some of the settings you have there to see
if fewer but larger threads work better than multiple smaller threads.

Tom

On Mon, Mar 22, 2021 at 10:52 AM Blake Carver 
wrote:

> I did some experimenting this weekend, messing around with indexer speeds,
> and found I could get it to succeed with the right indexer settings. I
> think the answer is going to be "it depends" and you'll need to experiment
> with what works on your set up with your data. I started with the defaults,
> then dropped it to realy slow (1 thread 1 per), then just tried to dial
> it up and down. The last one I tried worked fine, it was fast enough to
> finish in a reasonable amount of time and didn't slow down or crash. Your
> settings may not look like this, but here's something to try.
>
> AppConfig[:pui_indexer_records_per_thread] = 50
> AppConfig[:pui_indexer_thread_count] = 1
>
>
> So some extra detail for the mailing list archives... if your site keeps
> crashing before the indexers finish and you're not seeing any particular
> errors in the logs that make you think you have a problem with your data,
> try turning the knobs on your indexer speed and see if that helps.
>
> It looks like maybe the indexer just eats up too much memory on BIG
> records and having too many (too many being 15ish) threads running causes
> it to crash. I know BIG is pretty subjective, if you have a bunch of
> resources (maybe a few thousand) AND those resources all have ALLOTA (maybe
> a few thousand) children with ALLOTA subjects/agents/notes/stuff, then you
> might hit this problem. Seems like it's not the total number of resources,
> it's probably because those resources are big/complex/deep.
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Thursday, March 18, 2021 11:24 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] PUI indexing issues
>
> Dave,
>
> Thanks for the suggestion, but unless there is some direct limitation
> within Solr, that should not be an issue. My disk is at only about 50% of
> capacity and Solr should be able to expand as needed. In my case, I don't
> think there has been much addition to Solr because I'm reindexing records
> which have been indexed already. So the deleted records are growing, but
> not the overall number of records. My index is currently at about 6GB.
>
> Any other thoughts out there?
>
> Thanks,
> Tom
>
> On Thu, Mar 18, 2021 at 10:51 AM Mayo, Dave  wrote:
>
> This is a little bit of a shot in the dark, but have you looked at disk
> space on whatever host Solr is resident on? (the ASpace server if you’re
> not running an external one)?
>
> A thing we’ve hit a couple times is that Solr, at least in some
> configurations, needs substantial headroom on disk to perform well – I
> think it’s related to how it builds and maintains the index?  So it might
> be worth looking to see if Solr is filling up the disk enough that it can’t
> efficiently handle itself.
>
>
>
> --
>
> Dave Mayo (he/him)
>
> Senior Digital Library Software Engineer
> Harvard University > HUIT > LTS
>
>
>
> *From: * on
> behalf of Tom Hanstra 
> *Reply-To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Date: *Wednesday, March 17, 2021 at 11:43 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] PUI indexing issues
>
>
>
>
>
>
>
> - What really bothers me is the slowdown. That indicates to me that some
> resource is being lost along the way. Anyone have thoughts on what that
> might be?
>
>
>
>
>
> Just to follow up on my earlier post, I did get even lower numbers from
> Blake to try based upon what he used for our hosted account. But I'm seeing
> the same pattern in terms of slowdowns regarding the number of records that
> get processed/hour. Is this typical?  Is it just hitting records that have
> more work to be done? Or do I still have a resource issue.
>
>
>
> I note that the number of docs in Solr has not changed at all 

[Archivesspace_Users_Group] Logs and database connections

[Tom Hanstra]

Back again...

I have yet another wrinkle to things which I'm trying to track down.

I've started working with another instance of ArchivesSpace just to try to
move things forward a bit faster and now have some different issues going
on, this time with my database connections.

In the logs, things were humming along when, for reasons unknown, they no
longer were humming:

I, [2021-03-19T05:09:13.304429 #13392]  INFO -- : Thread-2868: Staff
Indexer [2021-03-19 05:09:13 -0400] ~~~ Indexed 595
420 of 763368 archival_object records in repository UNDA
I, [2021-03-19T05:09:14.625595 #13392]  INFO -- : Thread-2906: PUI Indexer
[2021-03-19 05:09:14 -0400] ~~~ Indexed 43278
0 of 763368 archival_object records in repository UNDA
E, [2021-03-19T05:09:14.768619 #13392] ERROR -- : Thread-5364: Failed
fetching archival_object id=809925: {"error":{"db_
error":["Database integrity constraint conflict:
Java::JavaSql::SQLNonTransientConnectionException: No operations allowe
d after connection closed."]}}

It *looks* like something got lost, but after some digging, I'm beginning
to wonder what is actually going on. Note that the PUI and Staff indexing
threads are 2906 and 2868 respectively. Throughout the log, this is the
case. Then, the thread which complains about connections is 5364...a
completely different thread. I see no other references to that thread in
any logs. So, what, exactly, timed out here?

The thing that has me scratching my head is that Solr is continuing to see
increased numbers of records added (I started that from scratch along with
everything else). So, how can records be being added to the Solr index
without showing up in the logs?  Or could it be that it is just the log
writer which is timed out but indexing is still going on in the
background?  Is there any way, besides the logs, to know where indexing
stands?

I note that this database was dedicated only to this instance of
ArchivesSpace and continues to see activity. So if things have truly
failed, what is it doing?

I have a feeling that indexing is still going on but the logging just
stopped working properly. But is there any way to tell?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

Dave,

Thanks for the suggestion, but unless there is some direct limitation
within Solr, that should not be an issue. My disk is at only about 50% of
capacity and Solr should be able to expand as needed. In my case, I don't
think there has been much addition to Solr because I'm reindexing records
which have been indexed already. So the deleted records are growing, but
not the overall number of records. My index is currently at about 6GB.

Any other thoughts out there?

Thanks,
Tom

On Thu, Mar 18, 2021 at 10:51 AM Mayo, Dave  wrote:

> This is a little bit of a shot in the dark, but have you looked at disk
> space on whatever host Solr is resident on? (the ASpace server if you’re
> not running an external one)?
>
> A thing we’ve hit a couple times is that Solr, at least in some
> configurations, needs substantial headroom on disk to perform well – I
> think it’s related to how it builds and maintains the index?  So it might
> be worth looking to see if Solr is filling up the disk enough that it can’t
> efficiently handle itself.
>
>
>
> --
>
> Dave Mayo (he/him)
>
> Senior Digital Library Software Engineer
> Harvard University > HUIT > LTS
>
>
>
> *From: * on
> behalf of Tom Hanstra 
> *Reply-To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Date: *Wednesday, March 17, 2021 at 11:43 AM
> *To: *Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject: *Re: [Archivesspace_Users_Group] PUI indexing issues
>
>
>
>
>
>
>
> - What really bothers me is the slowdown. That indicates to me that some
> resource is being lost along the way. Anyone have thoughts on what that
> might be?
>
>
>
>
>
> Just to follow up on my earlier post, I did get even lower numbers from
> Blake to try based upon what he used for our hosted account. But I'm seeing
> the same pattern in terms of slowdowns regarding the number of records that
> get processed/hour. Is this typical?  Is it just hitting records that have
> more work to be done? Or do I still have a resource issue.
>
>
>
> I note that the number of docs in Solr has not changed at all throughout
> the last couple of attempts, which again leads me to believe it has already
> handled these records (at least once) before and thus there is no more
> indexing to really be done with the records which it is running through
> the PUI indexer again. Which leads back to the "why does PUI indexing
> restart each time from 0" question. How does one add an enhancement request
> to have this reviewed and (perhaps) changed?
>
>
>
> Thanks,
>
> Tom
>
>
>
> --
>
> *Tom Hanstra*
>
> *Sr. Systems Administrator*
>
> hans...@nd.edu
>
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

>
>
> - What really bothers me is the slowdown. That indicates to me that some
> resource is being lost along the way. Anyone have thoughts on what that
> might be?
>
>
> Just to follow up on my earlier post, I did get even lower numbers from
Blake to try based upon what he used for our hosted account. But I'm seeing
the same pattern in terms of slowdowns regarding the number of records that
get processed/hour. Is this typical?  Is it just hitting records that have
more work to be done? Or do I still have a resource issue.

I note that the number of docs in Solr has not changed at all throughout
the last couple of attempts, which again leads me to believe it has already
handled these records (at least once) before and thus there is no more
indexing to really be done with the records which it is running through
the PUI indexer again. Which leads back to the "why does PUI indexing
restart each time from 0" question. How does one add an enhancement request
to have this reviewed and (perhaps) changed?

Thanks,
Tom

>
>> --
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

Thanks for the suggestion, Blake. A couple additional questions:

- Staff indexing is done and has its files written. So does the number of
threads given to that make a difference? Is it still taking up resources?

- Does there happen to be any way to stop the staff indexing and just let
PUI have full access to the server for indexing?

- What really bothers me is the slowdown. That indicates to me that some
resource is being lost along the way. Anyone have thoughts on what that
might be?

- Should our repositories be broken up into smaller groupings?  I'm
beginning to wonder if we have things set up incorrectly, since it sounds
like we have a very large data set compared to others.


And a comment

It is really frustrating to have to start over on the indexing each time.
It seems that there should be some way to document progress along the way
so that the indexing can pick up where it left off. Is that something that
might also be looked at?

Thanks all. Appreciate your help.

Tom


On Tue, Mar 16, 2021 at 1:15 PM Blake Carver 
wrote:

> > I've now left my PUI indexing threads and count at the default (which I
> believe is 1 thread and 25 records/thread).
>
> Try dropping both indexer_records_per_thread and indexer_thread_count for
> both PUI and Staff indexers. Maybe in half or so. Sometimes with larger
> records it just needs to be slowed down.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Tuesday, March 16, 2021 12:51 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] PUI indexing issues
>
> Hello again.
>
> I'm still trying to understand some indexing issues. I've now left my PUI
> indexing threads and count at the default (which I believe is 1 thread and
> 25 records/thread). And I have given 4GB to Java processes. I've tried
> other values as well, but with similar results.
>
> No matter what values I use, I cannot seem to fully index PUI. Each time,
> it will start well but continuously slow down. I've kept a spreadsheet of
> the number of records/hr I'm indexing and have several attempts which start
> in the 50-60K/hr range and then continuously slow down to the 1800-1500/hr
> speed until finally dying with a Java Heap error. I think I'm headed to
> that again this round.
>
> Why might this be happening?  Could my data have been corrupted during the
> transfer from Lyrasis? (I'm working with a database export of our
> production data). Is the database too far away (our database is in an AWS
> RDS being accessed from our AWS EC2).
>
> I do have one log which gave this error:
>
> E, [2021-03-12T18:14:53.886243 #2919] ERROR -- : Thread-9472: Failed
> fetching archival_object id=1484623: too many connection resets (due to
> Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 3150, last used
> 1615590893.870297 seconds
> ago
>
> prior to the Java Heap error. In that log, there were a number of
> connections for the staff indexer after the PUI indexer stopped reporting,
> then an 88 minute gap prior to the above connection error and then finally
> a Java Heap error in the archivesspace.out log.
>
> Does the indexer reauthenticate each time it connects to get more
> information?  The earlier question about authentication has me wondering if
> my database server might be balking at the number of reconnections or
> something. I'm trying to index 760K records.
>
> Bottom line is that I'm still not getting my PUI index creation to
> complete. Each run can take several days before it finally fails and I have
> to start all over again.  I'm looking for any help to track down why this
> slowdown is occurring and what I can do to address it.
>
> Thanks,
> Tom
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] PUI indexing issues

[Tom Hanstra]

Hello again.

I'm still trying to understand some indexing issues. I've now left my PUI
indexing threads and count at the default (which I believe is 1 thread and
25 records/thread). And I have given 4GB to Java processes. I've tried
other values as well, but with similar results.

No matter what values I use, I cannot seem to fully index PUI. Each time,
it will start well but continuously slow down. I've kept a spreadsheet of
the number of records/hr I'm indexing and have several attempts which start
in the 50-60K/hr range and then continuously slow down to the 1800-1500/hr
speed until finally dying with a Java Heap error. I think I'm headed to
that again this round.

Why might this be happening?  Could my data have been corrupted during the
transfer from Lyrasis? (I'm working with a database export of our
production data). Is the database too far away (our database is in an AWS
RDS being accessed from our AWS EC2).

I do have one log which gave this error:

E, [2021-03-12T18:14:53.886243 #2919] ERROR -- : Thread-9472: Failed
fetching archival_object id=1484623: too many connection resets (due to
Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 3150, last used
1615590893.870297 seconds
ago

prior to the Java Heap error. In that log, there were a number of
connections for the staff indexer after the PUI indexer stopped reporting,
then an 88 minute gap prior to the above connection error and then finally
a Java Heap error in the archivesspace.out log.

Does the indexer reauthenticate each time it connects to get more
information?  The earlier question about authentication has me wondering if
my database server might be balking at the number of reconnections or
something. I'm trying to index 760K records.

Bottom line is that I'm still not getting my PUI index creation to
complete. Each run can take several days before it finally fails and I have
to start all over again.  I'm looking for any help to track down why this
slowdown is occurring and what I can do to address it.

Thanks,
Tom
-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] PUI indexing

[Tom Hanstra]

Is it typical for PUI indexing to run through all records each time
ArchivesSpace restarts?  Or is my indexing just never fully completed?

The staff indexer seems to understand that there are no new records and
does not run through everything:

I, [2021-03-15T10:25:16.599063 #6928]  INFO -- : Thread-2868: Staff Indexer
[2021-03-15 10:25:16 -0400] Running index round
I, [2021-03-15T10:25:18.785671 #6928]  INFO -- : Thread-2868: Staff Indexer
[2021-03-15 10:25:18 -0400] Index round complete

But the PUI, so far in my experience, restarts from 0 each time. Just
trying to understand if this is normal behavior. I don't have anything in
the directory:

archivesspace/data/indexer_pui_state

besides the repositories_repositories.dat. Will there be more files there
when PUI indexing really completes?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

[Tom Hanstra]

Thanks, Megan. That update did fix things.

So, is this not something that can be addressed at the software fix level?
Do other sites simply stop indexing or turn something else off when we hit
DST in March?

Tom

On Mon, Mar 15, 2021 at 10:12 AM Schanz, Megan  wrote:

> I run across this every March for daylight savings. This is what I have in
> my notes to do each time if the indexer is running. Luckily it seems like I
> didn't have this scenario this year.
>
> java.lang.IllegalArgumentException: HOUR_OF_DAY: 2 ->
> 3:Java::JavaLang::IllegalArgumentException:
>
> This error will appear in the logs when trying to start ArchivesSpace
> after daylight savings time. Reference:
> http://lyralists.lyrasis.org/mailman/htdig/archivesspace_users_group/2019-March/006652.html
>
> Basically, the indexer user is being updated in the database with a time
> that does not exist due to daylight savings time.
>
> To verify:
>
> SELECT * FROM user WHERE (user_mtime >= '2021-03-14 02:00:00' and user_mtime 
> <= '2021-03-14 03:00:00') OR (system_mtime >= '2021-03-04 02:00:00' and 
> system_mtime <= '2021-03-14 03:00:00');
>
> The record that should come back is the search_indexer user.
>
> To resolve:
>
> UPDATE user set user_mtime = NOW(), system_mtime=NOW() where 
> username='search_indexer';
>
>
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Tom
> Hanstra 
> *Sent:* Monday, March 15, 2021 9:10 AM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Java error -
> Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
>
> Any suggestion on what ArchivesSpace might have changed?  I had the server
> running but indexing was complete. What might it have been changing and in
> what database table would I look for that change?
>
> Alternately, since this is still test data, should I just overlay a backup
> copy of the database from earlier than the EDT cutover time?  I don't know
> what affect that might have on other portions of ArchivesSpace.
>
> Finally, what is the overall fix for this issue. If others have seen it,
> what can be done to make sure to avoid it in the future?
>
> Thanks,
> Tom
>
> On Mon, Mar 15, 2021 at 9:03 AM Blake Carver 
> wrote:
>
> https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05
> <https://urldefense.com/v3/__https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05__;!!HXCxUKc!lsrL41tXx-WGwpdlFWm60sfuIbz0DRfuyn4RvaCAka7R1eaKVNkWL7PbhWHEm8g$>
>
> You'll want to find the date more like '2021-03-14 02:00%' or would it be 
> 03-13?
> Whatever the date was this year.
> Find any date with a time between 2-3am and just change it to any real
> hour.
> ArchivesSpace did "something" (probably restarted?) at a bad time on
> Sunday morning and wrote a time that never happened.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> James Bullen 
> *Sent:* Sunday, March 14, 2021 11:04 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Java error -
> Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
>
>
> Oh gee, I’ve seen that horror. And, yes, it is caused by records having
> times in that non-existent hour. From memory the only fix I could find was
> to delete the offending rows, or update their times.
>
>
> Cheers,
> James
>
>
>
> On Mar 15, 2021, at 1:59 PM, Tom Hanstra  wrote:
>
> I don't seem to be able to win with ArchivesSpace.
>
> After having indexing again finally give up because of another Java heap
> error, I restarted archivesspace again with some different java settings. I
> believe it actually got through the indexing, at least of the first
> repository.
>
> But tonight, looking at the logs, there is another error that seems to
> have something to do with, perhaps, daylight savings time?  I'm seeing this
> Java error in the logs, even after a restart:
>
> 
> INFO: An exception happened during JRuby-Rack startup
> Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
> --- System
> jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
> 25.272-b10 on 1.8.0_272-b10 +jit [linux-x86_64]
> Time: 2021-03-14 22:46:48 -0400
> Server: jetty/8.1.5.v20120716
> jruby.home: uri:classloader://META-INF/jruby.home
>
> --- Context Init Parameters:
> jruby

Re: [Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

[Tom Hanstra]

Any suggestion on what ArchivesSpace might have changed?  I had the server
running but indexing was complete. What might it have been changing and in
what database table would I look for that change?

Alternately, since this is still test data, should I just overlay a backup
copy of the database from earlier than the EDT cutover time?  I don't know
what affect that might have on other portions of ArchivesSpace.

Finally, what is the overall fix for this issue. If others have seen it,
what can be done to make sure to avoid it in the future?

Thanks,
Tom

On Mon, Mar 15, 2021 at 9:03 AM Blake Carver 
wrote:

> https://gist.github.com/Blake-/d493da28be5554a49a3a3835bbd98f05
>
> You'll want to find the date more like '2021-03-14 02:00%' or would it be 
> 03-13?
> Whatever the date was this year.
> Find any date with a time between 2-3am and just change it to any real
> hour.
> ArchivesSpace did "something" (probably restarted?) at a bad time on
> Sunday morning and wrote a time that never happened.
> --
> *From:* archivesspace_users_group-boun...@lyralists.lyrasis.org <
> archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of
> James Bullen 
> *Sent:* Sunday, March 14, 2021 11:04 PM
> *To:* Archivesspace Users Group <
> archivesspace_users_group@lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] Java error -
> Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
>
>
> Oh gee, I’ve seen that horror. And, yes, it is caused by records having
> times in that non-existent hour. From memory the only fix I could find was
> to delete the offending rows, or update their times.
>
>
> Cheers,
> James
>
>
>
> On Mar 15, 2021, at 1:59 PM, Tom Hanstra  wrote:
>
> I don't seem to be able to win with ArchivesSpace.
>
> After having indexing again finally give up because of another Java heap
> error, I restarted archivesspace again with some different java settings. I
> believe it actually got through the indexing, at least of the first
> repository.
>
> But tonight, looking at the logs, there is another error that seems to
> have something to do with, perhaps, daylight savings time?  I'm seeing this
> Java error in the logs, even after a restart:
>
> 
> INFO: An exception happened during JRuby-Rack startup
> Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
> --- System
> jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
> 25.272-b10 on 1.8.0_272-b10 +jit [linux-x86_64]
> Time: 2021-03-14 22:46:48 -0400
> Server: jetty/8.1.5.v20120716
> jruby.home: uri:classloader://META-INF/jruby.home
>
> --- Context Init Parameters:
> jruby.max.runtimes = 1
> jruby.min.runtimes = 1
> public.root = /
> rack.env = production
>
> --- Backtrace
> Sequel::DatabaseError: Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
>
> 
>
> Did something get corrupted in the database?  What might be happening to
> result in this error?
> How do I fix it?
>
> Thanks,
> Tom
>
> --
> *Tom Hanstra*
> *Sr. Systems Administrator*
> hans...@nd.edu
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>
> ___
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group@lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>


-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Java error - Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3

[Tom Hanstra]

I don't seem to be able to win with ArchivesSpace.

After having indexing again finally give up because of another Java heap
error, I restarted archivesspace again with some different java settings. I
believe it actually got through the indexing, at least of the first
repository.

But tonight, looking at the logs, there is another error that seems to have
something to do with, perhaps, daylight savings time?  I'm seeing this Java
error in the logs, even after a restart:


INFO: An exception happened during JRuby-Rack startup
Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3
--- System
jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 OpenJDK 64-Bit Server VM
25.272-b10 on 1.8.0_272-b10 +jit [linux-x86_64]
Time: 2021-03-14 22:46:48 -0400
Server: jetty/8.1.5.v20120716
jruby.home: uri:classloader://META-INF/jruby.home

--- Context Init Parameters:
jruby.max.runtimes = 1
jruby.min.runtimes = 1
public.root = /
rack.env = production

--- Backtrace
Sequel::DatabaseError: Java::JavaSql::SQLException: HOUR_OF_DAY: 2 -> 3



Did something get corrupted in the database?  What might be happening to
result in this error?
How do I fix it?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

[Archivesspace_Users_Group] Understanding why ArchivesSpace seems to have died

[Tom Hanstra]

I never was able to complete my indexing of PUI and now it looks as if
ArchivesSpace has simply stopped working for me. I'm trying to understand
what happened.

I have tried to break out the various logs, though doing so has not worked
as expected.  All PUI activity was going to the indexer log and the end of
that log shows the staff log only:

I, [2021-03-12T16:46:07.137484 #2919]  INFO -- : Thread-2868: Staff Indexer
[2021-03-12 16:46:07 -0500] Index round complete
I, [2021-03-12T16:46:40.315017 #2919]  INFO -- : Thread-2868: Staff Indexer
[2021-03-12 16:46:40 -0500] Running index round

I have to go back farther to see the last PUI update:

I, [2021-03-12T15:37:18.971100 #2919]  INFO -- : Thread-2932: PUI Indexer
[2021-03-12 15:37:18 -0500] ~~~ Indexed 760150 of 763368 archival_object
records in repository UNDA
I, [2021-03-12T15:41:27.314764 #2919]  INFO -- : Thread-2932: PUI Indexer
[2021-03-12 15:41:27 -0500] ~~~ Indexed 760200 of 763368 archival_object
records in repository UNDA

At the end, that index had gone to a crawl, only indexing about 400
records/hour.

While the indexer log continued through the 16:46 time stamp, the pui and
archivespace.out logs both stop at 16:10. Here is the last thing in the PUI
log:

I, [2021-03-12T16:10:06.564227 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered shared/_header.html.erb
(14.3ms)
I, [2021-03-12T16:10:06.570258 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered
shared/_navigation.html.erb (4.6ms)
I, [2021-03-12T16:10:06.572299 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9]   Rendered shared/_footer.html.erb
(0.4ms)
I, [2021-03-12T16:10:06.573882 #2919]  INFO -- :
[06eac683-aff7-4afe-9046-97e5551b94a9] Completed 404 Not Found in 2930ms
(Views: 79.4ms)

and in the archivesspace.out log:

INFO: [collection1] webapp= path=/update params={}
{add=[/repositories/2/archival_objects/1484673#pui,
/repositories/2/archival_objects/1484674#pui,
/repositories/2/archival_objects/1484675#pui,
/repositories/2/archival_objects/1484676#pui,
/repositories/2/archival_objects/1484677#pui,
/repositories/2/archival_objects/1484678#pui,
/repositories/2/archival_objects/1484679#pui,
/repositories/2/archival_objects/1484680#pui,
/repositories/2/archival_objects/1484681#pui,
/repositories/2/archival_objects/1484682#pui, ... (50 adds)]} 0 48
Mar 12, 2021 3:42:27 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
Mar 12, 2021 3:42:27 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/home/app/archivesspace/data/solr_index/index
lockFactory=NativeFSLockFactory@/home/app/archivesspace/data/solr_index/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_bx5,generation=15449}
commit{dir=NRTCachingDirectory(MMapDirectory@/home/app/archivesspace/data/solr_index/index
lockFactory=NativeFSLockFactory@/home/app/archivesspace/data/solr_index/index;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_bx6,generation=15450}
Mar 12, 2021 3:42:27 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit generation = 15450
Mar 12, 2021 3:42:27 PM org.apache.solr.search.SolrIndexSearcher 
INFO: Opening Searcher@20bc5398[collection1] realtime
Mar 12, 2021 3:42:27 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Mar 12, 2021 4:10:06 PM org.apache.solr.core.SolrCore execute
INFO: [collection1] webapp= path=/select
params={mm=100%25=(uri:("\/repositories\/\-1\/actuator\/\-1"))=\=true=0="=-exclude_by_default:true=publish:true=types:pui=1=json=true}
hits=0 status=0 QTime=140

Neither the Staff nor Public interface are currently responding right now.

How can I tell if this fully died and, if it did, why it died?  Should I
restart?

Thanks for any help anyone might have. This is rather frustrating!

Tom



-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Indexing and search issues

[Tom Hanstra]

On Thu, Mar 11, 2021 at 1:25 PM Andrew Morrison <
andrew.morri...@bodleian.ox.ac.uk> wrote:

> *> I also notice that indexing overall slows down as it gets farther into
> our records.*
>
> I haven't observed a slow-down in indexing. At least not noticeably enough
> to cause me to want to measure it. As I mentioned, there are some latter
> stages of the indexing process (building trees, committing changes) that
> can run for a long time without logging anything. But for the main indexing
> of archival objects, the last 1000 doesn't seem to take longer than the
> first 1000.
>
*I really only have one set of data to work from, but I've been tracking
times for PUI indexing especially. During the first hour, it was able to
complete193300 records. I'm assuming it was able to do that because it was
covering ground it had passed before the Java Heap error caused the
previous attempt to stop. But looking later in the processing, hour 3 it
indexed 41500 records, hour 12 it was down to 22250 records, and now in the
last hour (hour 23) it has only completed 950 records. That is why I was
asking about the slow down.*

> *> Is an external Solr a good idea for a site like ours?*
>
> Hard to say. There was a discussion on the performance benefits of running
> external Solr on here last month:
>
>
> http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2021-February/thread.html#8168
>
> And documentation here:
>
> https://archivesspace.github.io/tech-docs/provisioning/solr.html
> Whether it affects indexing speed for large numbers of records I cannot
> say. We made the decision to use it before all our data was migrated into
> ArchivesSpace.
>
> *Looks as if separating Solr out is a good idea, based upon feedback. I'll
work on that (assuming indexing finally finishes at some point) *

> Andrew.
>
>
> On 11/03/2021 15:32, Tom Hanstra wrote:
>
> Thanks, Andrew. Some responses intertwined below, italicized:
>
> On Thu, Mar 11, 2021 at 7:31 AM Andrew Morrison <
> andrew.morri...@bodleian.ox.ac.uk> wrote:
>
>> You can allocate more memory to ArchivesSpace by setting the
>> ASPACE_JAVA_XMX environment variable it runs under. Setting that to
>> "-Xmx4g" should be sufficient.
>>
> *I did bump that and the ASPACE_JAVA_XSS up a bit for this round, which
> looks like it will finally complete. Just a few more PUI records need to be
> added. *
>
>> Those FATAL lines in the log snippet are caused by a bot probing for
>> known vulnerabilities in common web platforms and applications, hoping to
>> find a web site running an out-of-date copy (of Drupal in this case) which
>> it can exploit. It has nothing to do with ArchivesSpace, which has no PHP
>> code. It is merely logging that it doesn't know what to do with that
>> request.
>>
> *Thanks. I was hoping this was just extraneous. *
>
>> How did you know your one successful re-indexing completed? There are two
>> indexers, Staff and PUI, with the latter usually taking much longer to
>> finish. So if the PUI indexer fails after the staff indexer finishes, you
>> will see more records in the staff interface than the public interface,
>> even if they're all set to be public. Also both indexers log messages that
>> could be interpreted as meaning they've finished, but they then run
>> additional indexing to build trees, to enable navigation within collections
>> to work. A finally they instruct Solr to commit changes, which can be slow
>> depending on the performance of your storage. You could try doubling
>> AppConfig[:indexer_solr_timeout_seconds] to allow more time for each
>> operation.
>>
>
>
> *At least one set of logs, when I earlier gave the server more resources,
> showed that the indexing had completed. But, because the second repository
> was showing nothing, I decided to indexing again.  This time around, we do
> have the second repository found so that, too, indicates that things have
> gone better. I guess I have to wait for things to complete but there are
> still some questions outstanding. For instance, one search I did for
> "football" (something dear to the Notre Dame experience) within the
> repository which is supposed to be pretty much indexed, showed over 32K
> results on our hosted site but only 17K locally. That seems wildly off with
> only a few PUI records to be completed (log shows 736500 of 763368). Could
> the incomplete index really be that far off?*
>
> *I also notice that indexing overall slows down as it gets farther into
> our records. Is that probably because there is just more to be done with
> the records that might not have gotten done in earlier attempts while the
> first records buzz

Re: [Archivesspace_Users_Group] Indexing and search issues

[Tom Hanstra]

Thanks, Andrew. Some responses intertwined below, italicized:

On Thu, Mar 11, 2021 at 7:31 AM Andrew Morrison <
andrew.morri...@bodleian.ox.ac.uk> wrote:

> You can allocate more memory to ArchivesSpace by setting the
> ASPACE_JAVA_XMX environment variable it runs under. Setting that to
> "-Xmx4g" should be sufficient.
>
*I did bump that and the ASPACE_JAVA_XSS up a bit for this round, which
looks like it will finally complete. Just a few more PUI records need to be
added. *

> Those FATAL lines in the log snippet are caused by a bot probing for known
> vulnerabilities in common web platforms and applications, hoping to find a
> web site running an out-of-date copy (of Drupal in this case) which it can
> exploit. It has nothing to do with ArchivesSpace, which has no PHP code. It
> is merely logging that it doesn't know what to do with that request.
>
*Thanks. I was hoping this was just extraneous. *

> How did you know your one successful re-indexing completed? There are two
> indexers, Staff and PUI, with the latter usually taking much longer to
> finish. So if the PUI indexer fails after the staff indexer finishes, you
> will see more records in the staff interface than the public interface,
> even if they're all set to be public. Also both indexers log messages that
> could be interpreted as meaning they've finished, but they then run
> additional indexing to build trees, to enable navigation within collections
> to work. A finally they instruct Solr to commit changes, which can be slow
> depending on the performance of your storage. You could try doubling
> AppConfig[:indexer_solr_timeout_seconds] to allow more time for each
> operation.
>


*At least one set of logs, when I earlier gave the server more resources,
showed that the indexing had completed. But, because the second repository
was showing nothing, I decided to indexing again. This time around, we do
have the second repository found so that, too, indicates that things have
gone better. I guess I have to wait for things to complete but there are
still some questions outstanding. For instance, one search I did for
"football" (something dear to the Notre Dame experience) within the
repository which is supposed to be pretty much indexed, showed over 32K
results on our hosted site but only 17K locally. That seems wildly off with
only a few PUI records to be completed (log shows 736500 of 763368). Could
the incomplete index really be that far off?*

*I also notice that indexing overall slows down as it gets farther into our
records. Is that probably because there is just more to be done with the
records that might not have gotten done in earlier attempts while the first
records buzz by rapidly because of earlier indexing attempts?  Or could it
be that resources are taken up early in the processing and no longer
available for processing the later records? Is resource tuning just a
trial/error prospect?  I don't see a lot of information in the
documentation.*

Or it could've re-indexed one repository but failed on the next. And it is
possible for entire repositories to be set as non-public, which could be
another explanation for fewer records.

Are you running an external Solr? If so, is the AppConfig[:solr_url] in
config.rb pointing to the correct server?
*I'm running a local Solr as part of the application. Is an external Solr a
good idea for a site like ours? I will also do some tweaking with the Solr
settings to see if that might help...after I get through at least one
complete index. *

> There are many possible reasons for search slowness, including not enough
> memory. Are there any differences in the speed of doing the same search in
> the staff and public interfaces? Or between two ways of getting the same
> results in the PUI. For example, does the link in the header to list all
> collections (/repositories/resources) return results faster than searching
> everything then filtering to just collections
> (/search?q[]=*[]=[]=keyword_fields[]=primary_type_values[]=resource).
> There's a fix coming in 3.0.0 for the latter.
>
*I had not tried comparing staff to public. I will do that (though I first
have to get some access to the staff side!). And I'll really not try to do
much comparison until we get indexing complete, in case the indexing itself
is slowing things down. *

*More questions to come, I'm sure. But thanks for your input and ideas of
places to look further. Much appreciated.*

> Andrew.
>
>
> On 11/03/2021 02:07, Tom Hanstra wrote:
>
> I'm very new to ArchivesSpace and so my issues may be early configuration
> problems. But I'm hoping some out there can assist. We are moving from
> hosted to local, so I have a large database full of data that I'm working
> with.
>
> Indexing
> Right now, I'm running into two primary problems:
>
> - Twice now, I've hit issues where the inde

[Archivesspace_Users_Group] Indexing and search issues

[Tom Hanstra]

I'm very new to ArchivesSpace and so my issues may be early configuration
problems. But I'm hoping some out there can assist. We are moving from
hosted to local, so I have a large database full of data that I'm working
with.

Indexing
Right now, I'm running into two primary problems:

- Twice now, I've hit issues where the indexing fails due to the Java heap
space being exhausted. Do others run into this? What do others use for Java
settings?
- I've broken out my PUI indexing log into a separate log and see FATAL
errors in the log:
--
I, [2021-03-10T15:32:03.747156 #2919]  INFO -- :
[1b34df32-d3b7-49c3-b205-01a59daf03e5] Started GET "/system_api.php"
 for 206.189.134.38 at 2021-03-10 15:32:03 -0500
F, [2021-03-10T15:32:03.881297 #2919] FATAL -- :
[1b34df32-d3b7-49c3-b205-01a59daf03e5]
F, [2021-03-10T15:32:03.881658 #2919] FATAL -- :
[1b34df32-d3b7-49c3-b205-01a59daf03e5] ActionController::RoutingErro
r (No route matches [GET] "/system_api.php"):
F, [2021-03-10T15:32:03.881866 #2919] FATAL -- :
[1b34df32-d3b7-49c3-b205-01a59daf03e5]
F, [2021-03-10T15:32:03.882085 #2919] FATAL -- :
[1b34df32-d3b7-49c3-b205-01a59daf03e5] actionpack (5.2.4.4) lib/acti
on_dispatch/middleware/debug_exceptions.rb:65:in `call'
[1b34df32-d3b7-49c3-b205-01a59daf03e5] actionpack (5.2.4.4)
lib/action_dispatch/middleware/show_exceptions.rb:33:in `
call'
--
Is this something to be concerned about? Why is it showing up in the PUI
log?

Search issues
- Supposedly, I did get one round of indexing completed without a heap
error. But the resulting searches yielded numbers which were incorrect
compared to our hosted version. This is why I've been trying reindexing. Is
it usual to have indexing *look* like it is complete but really be
incomplete?
- When I do a search, the response is really slow. I've got nginx set up as
a proxy in front of ArchivesSpace and it is showing that the slowness is in
ArchivesSpace itself somewhere. I don't see anything in the logs to show
what is taking so long. Where should I be checking for issues?

Thanks,
Tom

-- 
*Tom Hanstra*
*Sr. Systems Administrator*
hans...@nd.edu
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group