[Archivesspace_Users_Group] ArchivesSpace 3.4.0 now available

[Christine Di Bella]

Hello ArchivesSpace members,

ArchivesSpace is announcing the availability of v3.4.0. You can download it at 
https://github.com/archivesspace/archivesspace/releases/tag/v3.4.0.


This release includes many bug fixes and feature enhancements. Of particular 
note are some enhancements related to creating digital objects and making it 
easier to identify digital content for and in the public interface. The most 
complex work relates to new functionality using the previously unused Make 
Representative button. Based on a community-authored 
specification,
 it is now possible to mark a particular file version on a digital object or 
digital object component for display on different types of records, including 
digital objects, digital object components, accessions, resources, and archival 
objects. Representative images also display in relevant browse and search 
result displays in the public and staff interface.

Other enhancements for digital objects include the option to set a preference 
to spawn description fields from the linked accession, resource or archival 
object record when creating digital object instances; a View Digital Material 
section added to the "pill" for resource records in the public interface when 
there is digital content within the resource; breadcrumbs that make the context 
for linked digital objects easier to see in result displays; and a more 
recognizable generic button for digital content when an image cannot be shown. 
New documentation written by the User Documentation sub-team for many of the 
new features, including Make Representative, is available in the user 
manual.

Enhancements not related to digital objects but still significant include work 
submitted by Hudson Molonglo that it did for Princeton University to add local 
access restriction fields to the spreadsheet importers and RDE, and including 
URIs in EAD exports; more attractive print displays for records, including 
accessions; a new CSV import for standalone subjects; internal note fields for 
top containers and container profiles; and the elimination of a bug that caused 
hover text for the help center to sometimes appear at inopportune times.
Thanks to all of the community members who contributed code for this release: 
Andrew Morrison, Brian Harrington, Corey Schmidt, Dave Mayo, James Bullen, 
Jason Jordan, Jonathan Green, Joshua Shaw, Lora Woodford, Mark Triggs, Michael 
Glanznig, Nick Butler, and Payton Giles. As always, team members Brian Hoffman, 
Don Smith, Brian Zelip, and Mark Cooper, as well as contractor Manny Rodriguez, 
wrote and/or reviewed substantial code for the release. Thanks also to our 
community groups, especially the Testing sub-team, for their efforts.

More information about what's in the release and the link to download it are at 
https://github.com/archivesspace/archivesspace/releases/tag/v3.4.0. Information 
on upgrading to a new version of ArchivesSpace is available at 
https://archivesspace.github.io/tech-docs/administration/upgrading.html.

Due to the complex nature of some of the work in this release and the need for 
additional testing and refinement, this represents both the January and May 
releases from the 
roadmap.
 The next regular release of ArchivesSpace is projected for September.

Thanks as always for your support. If you have questions or need any assistance 
in upgrading, please let us know.

Christine

Christine Di Bella
ArchivesSpace Program Manager
christine.dibe...@lyrasis.org
800.999.8558 x2905
678-235-2905


[ASpaceOrgHomeMedium]

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace

[Peter Heiner]

Michael Smith wrote on 2023-05-23 23:52:45:
> Our team has been facing recurring issues with our ArchivesSpace setup since
> October last year, which we've been unable to fully resolve despite
> concerted efforts.

Could pretty much be a description of our team up until very recently, when a
team member was finally able to hook the application up with Datadog's tracing
facilities.

We're currently running 3.1.1 with a standalone Solr 7.7.3 and MariaDB 10.4.24
on Ubuntu 20.04/22.04 servers.

> The primary problem involves intermittent system slowdowns and shutdowns,
> requiring frequent reboots to regain functionality. This occurs on average
> 3-4 times weekly but can sometimes be more frequent. This issue is affecting
> multiple teams across our organization.

Using the tracing facilities mentioned above we've found that the object
resolver in Archivesspace does not deduplicate the object tree properly and as
a result a resource we had with over 1100 event links produced a 130MB+ JSON
object and was subsequently parsed into 1.3GB of Ruby data and due to a quirk
of rendering all this was done twice. We reported this on Github
(https://github.com/archivesspace/archivesspace/issues/2993) 3 weeks ago.
The events were not very important to our archivists, so we ended up deleting
them.

We've also found that search is also suboptimal for us. Searches are taking
exponentially longer with every added term and for every search thousands of
requests are made to populate the 'Found in' column of the results. We're on
an old version of Solr and are using a fairly old schema, so we want to
upgrade both before we report this issue.

We've also noticed that database queries trying to update the archivesspace
software agent's system_mtime are failing and we've found that the row has not
been updated since we switched from 2.8.1 to 3.1.1. Possibly linked to this...

> The most common symptom of our problem that we are seeing now looks to be a
> connection pool leak where what looks like indexer threads are holding
> connections in a closed wait state and preventing them from being used for
> other requests.  This leads to the main page timing out and staff seeing 504
> errors, when unresponsive in this manner we usually restart the application.

...our main problem: users are unable to save records due to the updates
timing out waiting for locks. Looking at the database processlist we've
observed 2-3 instances of identical update queries in different sessions and
on the tracing level the queries retry several times before failing on their
LIMIT 1 clause, as there are no rows to update. We don't fully understand this
problem yet, but seeing your message this might be because we don't see
indexer threads in the traces, as they're on a different host.

> Some of the things we’ve attempted so far,
> 
>   *   changed default config settings for indexer records per thread, thread
>   count and solr timeout to 10, 2 & 300


>   *   modified archivesspace.sh to increase memory available
>   (ASPACE_JAVA_XMX="-Xmx35g")

We're on 56GB of heap now. We have ~3.2 million objects in the database across
~30 repositories, I believe this to be one of the larger installations of AS
out there.

>   *   disabled both PUI and PUI indexer

We've been actually thinking of doing this, we currently have the indexer on a
separate host. Does disabling the indexers impact visibility of changes in any
way for you?

> Any advice with further diagnosis / troubleshooting would be appreciated. If
> you need additional information about our setup or the issues we're
> encountering, please let us know.

Our colleague has written a trivial plugin that enables Datadog tracing and
telemetry and it has been, excuse the phrasing, instrumental. He also made it
public, the brilliant bloke (use the log-scope branch for now):
https://gitlab.developers.cam.ac.uk/lib/dev/ams/aspace-datadog

Hope that helps,
p
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Excel Import Question

[Mendoza, David]

Thanks Valerie.  Maybe its my version or user error, but the importer is not 
taking action relative to the level I select for a resource.  I downloaded the 
attached file from the help site, but maybe its an older version.  It is a 
pretty cool tool for new adds though.

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Valerie 
Addonizio
Sent: Wednesday, May 24, 2023 12:43 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

David,

It is correct that that importer only creates new records, however they do not 
have to be for the collection and you can add children to series. The importer 
takes action at whatever level of the hierarchy is selected when you create the 
import job. If you're on a Resource record with no existing children, there is 
only one option (the resource) and in that case you will always get children of 
that resource; but if you're on a resource with a series, and you're on the 
series level when you click to import, the archival objects should be created 
under that series.

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Mendoza, David
Sent: Wednesday, May 24, 2023 12:22 PM
To: Archivesspace Users Group 
mailto:archivesspace_users_group@lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Two more question to all as I get more familiar.  I'm on AS version 3.0.1.
It looks like regardless of what hierarchy level I select while editing a 
collection, all imports seem to default to a child of the collection.  Is this 
true for everyone?
Secondly, is it correct to assume that regardless of the level of entry, the 
import sheet only creates new records?  Ie, it will not add to existing 
series/children already created in AS, correct?

Thanks!
David

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Valerie Addonizio
Sent: Tuesday, May 23, 2023 4:40 PM
To: Archivesspace Users Group 
mailto:archivesspace_users_group@lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Hi David, any record's URI is available at the top of the record when in View 
mode. Look for it next to the Created and Last Modified information.

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Mendoza, David
Sent: Tuesday, May 23, 2023 1:21 PM
To: 
archivesspace_users_group@lyralists.lyrasis.org
Subject: [Archivesspace_Users_Group] Excel Import Question

Hi All,
Apologies if this is a cross post.  I've been experimenting with the excel 
import sheet.  This is kind of embarrassing, but I cannot locate the Resource 
URI for a given resource/collection.  I assumed it was the Identifier at the 
collection level, but I'm getting errors that do not recognize the Resource ID. 
 Alternatively, I used an EAD ID and it worked, but the hierarchy codes are 
causing errors unless I designate an archival object as level 1.  I'm not sure 
if hierarchy codes only work if a Resource ID is used.

I'm not sure if this is an issue with the excel sheet or just user error.  If 
any one has any advice or a youtube clip (I couldn't find one), it would 
greatly be appreciated.

Thanks!
David

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.

--
This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended 
recipient, please delete this message.


bulk_import_template.xlsx
Description: bulk_import_template.xlsx
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasi

Re: [Archivesspace_Users_Group] Excel Import Question

[Valerie Addonizio]

David,

It is correct that that importer only creates new records, however they do not 
have to be for the collection and you can add children to series. The importer 
takes action at whatever level of the hierarchy is selected when you create the 
import job. If you're on a Resource record with no existing children, there is 
only one option (the resource) and in that case you will always get children of 
that resource; but if you're on a resource with a series, and you're on the 
series level when you click to import, the archival objects should be created 
under that series.

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Mendoza, 
David
Sent: Wednesday, May 24, 2023 12:22 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Two more question to all as I get more familiar.  I'm on AS version 3.0.1.
It looks like regardless of what hierarchy level I select while editing a 
collection, all imports seem to default to a child of the collection.  Is this 
true for everyone?
Secondly, is it correct to assume that regardless of the level of entry, the 
import sheet only creates new records?  Ie, it will not add to existing 
series/children already created in AS, correct?

Thanks!
David

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Valerie Addonizio
Sent: Tuesday, May 23, 2023 4:40 PM
To: Archivesspace Users Group 
mailto:archivesspace_users_group@lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Hi David, any record's URI is available at the top of the record when in View 
mode. Look for it next to the Created and Last Modified information.

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Mendoza, David
Sent: Tuesday, May 23, 2023 1:21 PM
To: 
archivesspace_users_group@lyralists.lyrasis.org
Subject: [Archivesspace_Users_Group] Excel Import Question

Hi All,
Apologies if this is a cross post.  I've been experimenting with the excel 
import sheet.  This is kind of embarrassing, but I cannot locate the Resource 
URI for a given resource/collection.  I assumed it was the Identifier at the 
collection level, but I'm getting errors that do not recognize the Resource ID. 
 Alternatively, I used an EAD ID and it worked, but the hierarchy codes are 
causing errors unless I designate an archival object as level 1.  I'm not sure 
if hierarchy codes only work if a Resource ID is used.

I'm not sure if this is an issue with the excel sheet or just user error.  If 
any one has any advice or a youtube clip (I couldn't find one), it would 
greatly be appreciated.

Thanks!
David

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Excel Import Question

[Mendoza, David]

Two more question to all as I get more familiar.  I'm on AS version 3.0.1.
It looks like regardless of what hierarchy level I select while editing a 
collection, all imports seem to default to a child of the collection.  Is this 
true for everyone?
Secondly, is it correct to assume that regardless of the level of entry, the 
import sheet only creates new records?  Ie, it will not add to existing 
series/children already created in AS, correct?

Thanks!
David

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Valerie 
Addonizio
Sent: Tuesday, May 23, 2023 4:40 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Hi David, any record's URI is available at the top of the record when in View 
mode. Look for it next to the Created and Last Modified information.

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Mendoza, 
David
Sent: Tuesday, May 23, 2023 1:21 PM
To: archivesspace_users_group@lyralists.lyrasis.org
Subject: [Archivesspace_Users_Group] Excel Import Question

Hi All,
Apologies if this is a cross post.  I've been experimenting with the excel 
import sheet.  This is kind of embarrassing, but I cannot locate the Resource 
URI for a given resource/collection.  I assumed it was the Identifier at the 
collection level, but I'm getting errors that do not recognize the Resource ID. 
 Alternatively, I used an EAD ID and it worked, but the hierarchy codes are 
causing errors unless I designate an archival object as level 1.  I'm not sure 
if hierarchy codes only work if a Resource ID is used.

I'm not sure if this is an issue with the excel sheet or just user error.  If 
any one has any advice or a youtube clip (I couldn't find one), it would 
greatly be appreciated.

Thanks!
David

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.

--
This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended 
recipient, please delete this message.
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace

[Joshua D. Shaw]

Hi Michael

These aren't answers, but I think it might help the group if we knew a bit more 
about how your instance is structured - both from a tech perspective (memory 
allocation to the app and Solr) and things like how many repos and how many 
objects (resources, AOs, etc) are in the DB. The structure of your resources 
may also be useful. IE are they wide or deep or both? Wide meaning a lot of 
siblings at each level, but not a lot of levels in the hierarchy and deep 
meaning a lot of levels in the hierarchy, but not as many siblings at each 
level.

The plugins that you are using probably aren't the culprit, but they can 
add/override index functionality, so listing those out may help as well.

It might also be good to know how many edits are made concurrently on average.

Couple of things that sprang to mind to check (if you haven't already). Have 
you noticed this same behavior in an instance that is not in use? IE have you 
set up a clone of your production instance, let it do its initial full index, 
and then just let it sit? Do you see errors in the app log that have any 
bearing on the problem or pop up around or just before the app goes 
unresponsive or OOM?

In case it helps for comparison, Dartmouth is running 3.3.1 (skipped 3.2.0) and 
allocating 4GB each to the app and Solr - everything running in containers. We 
have 5 repos, though only one is utilized much. That repo has about 15k 
resources and 670k AOs with 30k top containers and 15k agents. We have 
relatively few events or subjects. The resources tend to be wide with max 4 
levels of hierarchy. Our largest resource has 10s of thousands of AOs in the 
hierarchy. We also run a huge number of plugins. We have relatively few editors 
- less than 5 at any one time.

Full index typically takes about 24 hours. We have not seen memory issues in 
any of our instances, though I have occasionally seen indexer timeouts during a 
full index. We have stock settings for the indexer (4, 1, 25) - though I had to 
raise the solr timeout a huge amount to 7200 for 3.3.1 to avoid solr timeouts. 
We do run the PUI, so much of the full index time is the PUI index churning 
away. Staff side indexing takes about 6-8 hours.

Best,
Joshua


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Michael 
Smith 
Sent: Tuesday, May 23, 2023 7:52 PM
To: archivesspace_users_group@lyralists.lyrasis.org 

Subject: [Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace

You don't often get email from mism...@nla.gov.au. Learn why this is 
important

Hello,



Our team has been facing recurring issues with our ArchivesSpace setup since 
October last year, which we've been unable to fully resolve despite concerted 
efforts.



We’re currently running v3.2 on Red Hat Enterprise Linux Server 7.9 (Maipo) and 
we do have a few custom plugins developed by Hudmol. These don’t appear to be 
causing the issues that we’re seeing but we haven’t ruled that out yet.



The primary problem involves intermittent system slowdowns and shutdowns, 
requiring frequent reboots to regain functionality. This occurs on average 3-4 
times weekly but can sometimes be more frequent. This issue is affecting 
multiple teams across our organization.



The most common symptom of our problem that we are seeing now looks to be a 
connection pool leak where what looks like indexer threads are holding 
connections in a closed wait state and preventing them from being used for 
other requests.  This leads to the main page timing out and staff seeing 504 
errors, when unresponsive in this manner we usually restart the application. If 
the application hits an OOM, it will restart itself.



Some of the things we’ve attempted so far,



  *   changed default config settings for indexer records per thread, thread 
count and solr timeout to 10, 2 & 300
  *   modified archivesspace.sh to increase memory available 
(ASPACE_JAVA_XMX="-Xmx35g")
  *   disabled both PUI and PUI indexer
  *   application logging to a circular log
  *   changed the garbage collection policies 
(ASPACE_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC 
-XX:NewRatio=1 -XX:+ExitOnOutOfMemoryError -XX:+UseGCOverheadLimit")
  *   checked top_containers with empty relationships (0 results)
  *   checked for duplicate event relationships (0 results)
  *   checked for empty indexer state files per record type (0 empty state 
files)
  *   nightly restarts of the system



Any advice with further diagnosis / troubleshooting would be appreciated. If 
you need additional information about our setup or the issues we're 
encountering, please let us know.



Regards,



Michael Smith  |  Software Developer
02 6262 1029  |  mism...@nla.gov.au  |  National 
Library of Australia

The National Library of Australia acknowledges Australia’s First Nations 
Peoples – the First Australians – as the Traditio

Re: [Archivesspace_Users_Group] Excel Import Question

[Mendoza, David]

Wonderful.  Thank you Valerie.

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Valerie 
Addonizio
Sent: Tuesday, May 23, 2023 4:40 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Excel Import Question

Hi David, any record's URI is available at the top of the record when in View 
mode. Look for it next to the Created and Last Modified information.

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Mendoza, 
David
Sent: Tuesday, May 23, 2023 1:21 PM
To: archivesspace_users_group@lyralists.lyrasis.org
Subject: [Archivesspace_Users_Group] Excel Import Question

Hi All,
Apologies if this is a cross post.  I've been experimenting with the excel 
import sheet.  This is kind of embarrassing, but I cannot locate the Resource 
URI for a given resource/collection.  I assumed it was the Identifier at the 
collection level, but I'm getting errors that do not recognize the Resource ID. 
 Alternatively, I used an EAD ID and it worked, but the hierarchy codes are 
causing errors unless I designate an archival object as level 1.  I'm not sure 
if hierarchy codes only work if a Resource ID is used.

I'm not sure if this is an issue with the excel sheet or just user error.  If 
any one has any advice or a youtube clip (I couldn't find one), it would 
greatly be appreciated.

Thanks!
David

This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended 
recipient, please delete this message.

--
This message, and any attachments, is for the intended recipient(s) only, may 
contain information that is privileged, confidential and/or proprietary and 
subject to important terms and conditions available at 
http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended 
recipient, please delete this message.
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group