Thanks Blake. A mass of solr logging (I am set to ERROR throughout, but still getting plenty of indexer INFO). Prior to that:
========== Scott Renton Digital Library Development & Systems Floor F East Argyle House 515219 ________________________________ From: [email protected] <[email protected]> on behalf of Blake Carver <[email protected]> Sent: 07 September 2021 15:37 To: [email protected] <[email protected]> Subject: Re: [Archivesspace_Users_Group] Connection/login dropping This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. What's happening in the logs just before that first "Unhanded exception" error? ________________________________ From: [email protected] <[email protected]> on behalf of RENTON Scott <[email protected]> Sent: Monday, September 6, 2021 9:29 AM To: [email protected] <[email protected]> Subject: [Archivesspace_Users_Group] Connection/login dropping Hi all Our uptime has been a bit flaky the last few weeks- what we're finding is that, after a certain amount of service, users are unable to login, and if that's not noticed or acted on, the site generally stops responding some time later. A restart of the application resolves it. We're running 2.8.1 against MariaDB 5.5-65, and Java 8. I have tried different versions of the mysqlconnector, but this does not seem to make a difference (first noticed on 5.1.39). The MySQL has been tuned to an extent. The VM has 16GB of memory, approximately 10GB of which is apportioned to AS. The MySQL process was restarted last week, and clearing that has still allowed the issue. It's pretty hard to spot any particular action which causes this. The logging we see once it's happened looks like this: E, [2021-08-18T06:21:01.548843 #90658] ERROR -- : Thread-28310: Unhandled exception! E, [2021-08-18T06:21:01.552325 #90658] ERROR -- : apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb:783) apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0$__VARARGS__(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) and will continue to spout "minus_5_dot_9_dot_0" type errors until we reboot. I have taken some statistics around the numbers of threads running pre and post bounce. I don't see anything obvious there, and I have left the config.rb value AppConfig[:db_max_connections] = proc { 20 + (AppConfig[:indexer_thread_count] * 2) } as it is. I'm also aware that we have an issue some others have of a proliferation of FATAL errors since merging top/subcontainers (items cannot be found), but I do not know if it is related to this. However, if anyone recognises this behaviour, and can advise, I'd be grateful to hear about it. Cheers Scott ========== Scott Renton Digital Library Development & Systems Floor F East Argyle House 515219 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
_______________________________________________ Archivesspace_Users_Group mailing list [email protected] http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
