Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
Yes, Trevor that would be great! Mark, I’ll look into whether we can try tweaking that xml_cleaner.rb file. That scenario you describe below is exactly what’s happening for us—although I should mention that we can still export valid EAD. For the AT example (which is in a Separated Materials note), AT gets changed to “ATT” in the EAD export, but PDF generation fails. If I change the actual text in the ArchivesSpace note to “ATT”, it still shows up as “ATT” in the exported EAD, but PDF generation works! Dave, I do remember there being a lot of HTML/XML entities in our title fields when we were using Archon, but I’m not seeing them in ArchivesSpace. Maybe we’ll do some more manual review before more seriously considering making a change on the SQL level. Thanks! --Benn From: archivesspace_users_group-boun...@lyralists.lyrasis.org [mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org] On Behalf Of Custer, Mark Sent: Friday, September 22, 2017 9:11 AM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org> Subject: Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface Trevor, That would be fantastic! Benn, For your PDF error, I think that might be caused by a slightly different issue. The new PUI “Print to PDF” process converts that ArchivesSpace JSON record to HTML and then converts that HTML into a PDF file. So, it doesn’t use the same JSON --> EAD --> PDF process as the staff interface. I’m assuming that a small tweak to this file https://github.com/archivesspace/archivesspace/blob/master/public/app/lib/xml_cleaner.rb<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_archivesspace_archivesspace_blob_master_public_app_lib_xml-5Fcleaner.rb=DwMGaQ=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws=fciHLC2ou0tXKp-JlPlsrEmslFw9tnR331DgXAhVLvo=g2hj41KrnOkI0qiFWudGx_9RW8-EmhNYJuivlbJ-lyI=HkfvJ7NeFHImxk6SoaPpQaXRTa32X41GrUeQG-HCib8=> might allow it to still create the PDF successfully (assuming that ArchivesSpace would want the application to handle both “b” and “bw”, which might not be the case). We should log this issue in JIRA at some point, regardless, just so that it’s captured there. I don’t have time to do that right now, but I did update one of the files in the sandbox to illustrate the problem. Here it is: http://public.archivesspace.org/repositories/2/resources/1008/<https://urldefense.proofpoint.com/v2/url?u=http-3A__public.archivesspace.org_repositories_2_resources_1008_=DwMGaQ=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws=fciHLC2ou0tXKp-JlPlsrEmslFw9tnR331DgXAhVLvo=g2hj41KrnOkI0qiFWudGx_9RW8-EmhNYJuivlbJ-lyI=q8VeN3DkWhfOKn4g1e02xDt-gxOL3kkw_ZCaHiIdajQ=> * Before I added the lone collection-level note to this record, the PDF printed fine. * Once I added a note of “b”, it failed. * When I change the note to “bw”, the PDF file works * It also still displays fine in the PUI, which might mean that the problem that I noted in my previous message only occurs when the note is in one of those “see more” / “see less” sections. Mark From: archivesspace_users_group-boun...@lyralists.lyrasis.org<mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org> [mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org] On Behalf Of Trevor Thornton Sent: Friday, 22 September, 2017 9:45 AM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org<mailto:archivesspace_users_group@lyralists.lyrasis.org>> Subject: Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface The logic for converting ampersands in the EAD exporter is to only convert them if they are immediately followed by a space, otherwise they are assumed to be an entity. This is part of the process of sanitizing mixed content, which is actually applied to most fields. However, the ampersand conversion is included in the routine that handles line breaks (converting 2 line breaks into paragraphs as appropriate), and this is only applied to fields for which the corresponding EAD tag allows as a child, which excludes untititle, abstract, etc. There's no good reason that I can think of why the ampersand conversion should be restricted in this way, so it can probably be moved to apply more broadly. Unfortunately, since the new EAD3 exporter is based on the existing EAD exporter, this problem persists in the EAD3 exporter, because I didn't really notice it until now. I'll try to fix it in both places and do a pull request. On Fri, Sep 22, 2017 at 8:48 AM, Mayo, Dave <dave_m...@harvard.edu<mailto:dave_m...@harvard.edu>> wrote: Hi Benn, This is a recurring issue I hit over both Harvard and Smith’s collections – it’s a consequence of ASpace not really having a distinction between mixed content and plaintext content. Unfortunately, there isn’t really a good solution. The best solution
Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
Trevor, That would be fantastic! Benn, For your PDF error, I think that might be caused by a slightly different issue. The new PUI “Print to PDF” process converts that ArchivesSpace JSON record to HTML and then converts that HTML into a PDF file. So, it doesn’t use the same JSON --> EAD --> PDF process as the staff interface. I’m assuming that a small tweak to this file https://github.com/archivesspace/archivesspace/blob/master/public/app/lib/xml_cleaner.rb might allow it to still create the PDF successfully (assuming that ArchivesSpace would want the application to handle both “b” and “bw”, which might not be the case). We should log this issue in JIRA at some point, regardless, just so that it’s captured there. I don’t have time to do that right now, but I did update one of the files in the sandbox to illustrate the problem. Here it is: http://public.archivesspace.org/repositories/2/resources/1008/ * Before I added the lone collection-level note to this record, the PDF printed fine. * Once I added a note of “b”, it failed. * When I change the note to “bw”, the PDF file works * It also still displays fine in the PUI, which might mean that the problem that I noted in my previous message only occurs when the note is in one of those “see more” / “see less” sections. Mark From: archivesspace_users_group-boun...@lyralists.lyrasis.org [mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org] On Behalf Of Trevor Thornton Sent: Friday, 22 September, 2017 9:45 AM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org> Subject: Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface The logic for converting ampersands in the EAD exporter is to only convert them if they are immediately followed by a space, otherwise they are assumed to be an entity. This is part of the process of sanitizing mixed content, which is actually applied to most fields. However, the ampersand conversion is included in the routine that handles line breaks (converting 2 line breaks into paragraphs as appropriate), and this is only applied to fields for which the corresponding EAD tag allows as a child, which excludes untititle, abstract, etc. There's no good reason that I can think of why the ampersand conversion should be restricted in this way, so it can probably be moved to apply more broadly. Unfortunately, since the new EAD3 exporter is based on the existing EAD exporter, this problem persists in the EAD3 exporter, because I didn't really notice it until now. I'll try to fix it in both places and do a pull request. On Fri, Sep 22, 2017 at 8:48 AM, Mayo, Dave <dave_m...@harvard.edu<mailto:dave_m...@harvard.edu>> wrote: Hi Benn, This is a recurring issue I hit over both Harvard and Smith’s collections – it’s a consequence of ASpace not really having a distinction between mixed content and plaintext content. Unfortunately, there isn’t really a good solution. The best solution as far as I’ve been able to figure is to use HTML/XML entity for ampersand () wherever it appears in a context that’s treated by the interface/etc as markup; title fields _definitely_ fall under that category. There’s unfortunately no reliable guide to what fields are “mixed content” and what fields are “plaintext content” because, well, the underlying system doesn’t track that distinction – it’s up to how the fields are eventually displayed/used to build exports/etc. As to _how_ to fix it – well, it depends somewhat on whether you can be ABSOLUTELY SURE you don’t have any HTML/XML entities in your title fields. If you are ABSOLUTELY SURE of this, you should be able to make the change via API or on the SQL level, but if you DO have entities, it gets a lot harder, to the point where manual review is probably appropriate. - Dave Mayo ASpace Core Committer’s Group Member From: <archivesspace_users_group-boun...@lyralists.lyrasis.org<mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>> on behalf of Benn Joseph <benn.jos...@northwestern.edu<mailto:benn.jos...@northwestern.edu>> Reply-To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org<mailto:archivesspace_users_group@lyralists.lyrasis.org>> Date: Thursday, September 21, 2017 at 4:21 PM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org<mailto:archivesspace_users_group@lyralists.lyrasis.org>> Subject: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface Hi all, We've encountered an issue with the v2.1.2 Print-to-PDF button in the public interface--apparently for any resource record with an ampersand that is followed immediately by another character that is not a space (e.g. "b" or "AT"), the ampersand is misinterpreted and causes the Print-to-PDF button to fail wit
Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
The logic for converting ampersands in the EAD exporter is to only convert them if they are immediately followed by a space, otherwise they are assumed to be an entity. This is part of the process of sanitizing mixed content, which is actually applied to most fields. However, the ampersand conversion is included in the routine that handles line breaks (converting 2 line breaks into paragraphs as appropriate), and this is only applied to fields for which the corresponding EAD tag allows as a child, which excludes untititle, abstract, etc. There's no good reason that I can think of why the ampersand conversion should be restricted in this way, so it can probably be moved to apply more broadly. Unfortunately, since the new EAD3 exporter is based on the existing EAD exporter, this problem persists in the EAD3 exporter, because I didn't really notice it until now. I'll try to fix it in both places and do a pull request. On Fri, Sep 22, 2017 at 8:48 AM, Mayo, Dave <dave_m...@harvard.edu> wrote: > Hi Benn, > > This is a recurring issue I hit over both Harvard and Smith’s collections > – it’s a consequence of ASpace not really having a distinction between > mixed content and plaintext content. > > Unfortunately, there isn’t really a good solution. The best solution as > far as I’ve been able to figure is to use HTML/XML entity for ampersand > () wherever it appears in a context that’s treated by the > interface/etc as markup; title fields _*definitely*_ fall under that > category. There’s unfortunately no reliable guide to what fields are > “mixed content” and what fields are “plaintext content” because, well, the > underlying system doesn’t track that distinction – it’s up to how the > fields are eventually displayed/used to build exports/etc. > > > As to _*how*_ to fix it – well, it depends somewhat on whether you can be > ABSOLUTELY SURE you don’t have any HTML/XML entities in your title fields. > If you are ABSOLUTELY SURE of this, you should be able to make the change > via API or on the SQL level, but if you DO have entities, it gets a lot > harder, to the point where manual review is probably appropriate. > > - Dave Mayo > ASpace Core Committer’s Group Member > > *From: *<archivesspace_users_group-boun...@lyralists.lyrasis.org> on > behalf of Benn Joseph <benn.jos...@northwestern.edu> > *Reply-To: *Archivesspace Users Group <archivesspace_users_group@ > lyralists.lyrasis.org> > *Date: *Thursday, September 21, 2017 at 4:21 PM > *To: *Archivesspace Users Group <archivesspace_users_group@ > lyralists.lyrasis.org> > *Subject: *[Archivesspace_Users_Group] ampersand issue with PDF button in > 2.1.2 public interface > > > > Hi all, > > We've encountered an issue with the v2.1.2 Print-to-PDF button in the > public interface--apparently for any resource record with an ampersand that > is followed immediately by another character that is not a space (e.g. > "b" or "AT"), the ampersand is misinterpreted and causes the > Print-to-PDF button to fail with an error. For me, that error is just > "something went wrong", but the log shows this (when it gets tripped up on > "b"): > > > > RuntimeError (Failed to clean XML: The reference to entity "w" must end > with the ';' delimiter.): > > > > So we're guessing ArchivesSpace is thinking "" should be "", and so > forth for any other string of text with an ampersand. I checked this by > going into a record that wouldn’t print and changing the lone suspect > ampersand (“AT” to “AT and T”) and the PDF generated just fine. > > > > This doesn't impact being able to just view resource records in the public > interface, it's just the PDF function that isn't working. It's a problem, > though, because we want to be able to use that PDF functionality but we > also have a lot of ampersands in our resource records! Has anyone else > experienced this issue or possibly come up with a fix? > > > > Thanks, > > --Benn > > > > *Benn Joseph* > > Head of Archival Processing > > Northwestern University Libraries > > Northwestern University > > www.library.northwestern.edu > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.library.northwestern.edu=DwMFAg=WO-RGvefibhHBZq3fL85hQ=_Mv1dY22K7jvT5MD7xjbvGVzRDOUMhx4WYcnPSIzYnE=m73cREghXWiIzy9ulXvIZW1Mx-NoJoH_rB1LSdzHQ6Q=Xj5cFVS13R-ioWYCsYqxItOviZziBf6vpg_FBhiC1c4=> > > benn.jos...@northwestern.edu <benn.jos...@northwestern.edu%0d> > > 847.467.6581 <(847)%20467-6581> > > > > ___ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group@lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries ___ Archivesspace_Users_Group mailing list Archivesspace_Users_Group@lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
Dave, Benn: Another important point here is that the ASpace staff interface attempts to handle both “ & “ and “ “. Those spaces are (quite unfortunately) important here. is a special HTML and XML entity reference for the ampersand character. A nice, short overview is provided here: https://mrcoles.com/blog/how-use-amersands-html-encode/ Benn, as you’ve discovered, when there aren’t any spaces, it looks like those attempts to handle both types of references go out the window. So, AT, b, , , , cause problems. The really problematic part is that the problems and the “solutions” vary depending on whether you’re exporting data, storing it titles vs. notes, displaying it in the PUI, creating a PDF, I just recently discovered another wrinkle that puts us into a real catch 22. Here it is: We’ve got a note with “” in ArchivesSpace. It would be great if that could be handled okay everywhere, whether ArchivesSpace forced us to use OR c (and I don’t care which way!), but as Dave explains, there aren’t clear distinctions for ASpace to make in every situation right now. With that “” in the note, here’s what I wind up with: * The PUI displays the note correctly. Yay! * The EAD exporter won’t wrap paragraph elements around the note, which results in invalid EAD. Boo. If I edit that note to instead be “c”, here’s what I wind up with: * The PUI displays everything in the note up until that character. Everything else in the note silently falls away into oblivion (but surely this would be an easy bug fix, I hope???). Boo. * The EAD exporter works perfectly. Yay! Dave, since you included the line about being on the Core Committer’s Group, does that mean that this issue will be discussed soon ☺ (I know that it wouldn’t be an easy thing to tackle, but it would make a lot of metadata folks out there pretty happy!) Mark From: archivesspace_users_group-boun...@lyralists.lyrasis.org [mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org] On Behalf Of Mayo, Dave Sent: Friday, 22 September, 2017 8:49 AM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org> Subject: Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface Hi Benn, This is a recurring issue I hit over both Harvard and Smith’s collections – it’s a consequence of ASpace not really having a distinction between mixed content and plaintext content. Unfortunately, there isn’t really a good solution. The best solution as far as I’ve been able to figure is to use HTML/XML entity for ampersand () wherever it appears in a context that’s treated by the interface/etc as markup; title fields _definitely_ fall under that category. There’s unfortunately no reliable guide to what fields are “mixed content” and what fields are “plaintext content” because, well, the underlying system doesn’t track that distinction – it’s up to how the fields are eventually displayed/used to build exports/etc. As to _how_ to fix it – well, it depends somewhat on whether you can be ABSOLUTELY SURE you don’t have any HTML/XML entities in your title fields. If you are ABSOLUTELY SURE of this, you should be able to make the change via API or on the SQL level, but if you DO have entities, it gets a lot harder, to the point where manual review is probably appropriate. - Dave Mayo ASpace Core Committer’s Group Member From: <archivesspace_users_group-boun...@lyralists.lyrasis.org<mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>> on behalf of Benn Joseph <benn.jos...@northwestern.edu<mailto:benn.jos...@northwestern.edu>> Reply-To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org<mailto:archivesspace_users_group@lyralists.lyrasis.org>> Date: Thursday, September 21, 2017 at 4:21 PM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org<mailto:archivesspace_users_group@lyralists.lyrasis.org>> Subject: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface Hi all, We've encountered an issue with the v2.1.2 Print-to-PDF button in the public interface--apparently for any resource record with an ampersand that is followed immediately by another character that is not a space (e.g. "b" or "AT"), the ampersand is misinterpreted and causes the Print-to-PDF button to fail with an error. For me, that error is just "something went wrong", but the log shows this (when it gets tripped up on "b"): RuntimeError (Failed to clean XML: The reference to entity "w" must end with the ';' delimiter.): So we're guessing ArchivesSpace is thinking "" should be "", and so forth for any other string of text with an ampersand. I checked this by going into a record that wouldn’t print and changing the lone suspect ampersand (“AT” to “AT and T”)
Re: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
Hi Benn, This is a recurring issue I hit over both Harvard and Smith’s collections – it’s a consequence of ASpace not really having a distinction between mixed content and plaintext content. Unfortunately, there isn’t really a good solution. The best solution as far as I’ve been able to figure is to use HTML/XML entity for ampersand () wherever it appears in a context that’s treated by the interface/etc as markup; title fields _definitely_ fall under that category. There’s unfortunately no reliable guide to what fields are “mixed content” and what fields are “plaintext content” because, well, the underlying system doesn’t track that distinction – it’s up to how the fields are eventually displayed/used to build exports/etc. As to _how_ to fix it – well, it depends somewhat on whether you can be ABSOLUTELY SURE you don’t have any HTML/XML entities in your title fields. If you are ABSOLUTELY SURE of this, you should be able to make the change via API or on the SQL level, but if you DO have entities, it gets a lot harder, to the point where manual review is probably appropriate. - Dave Mayo ASpace Core Committer’s Group Member From: <archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Benn Joseph <benn.jos...@northwestern.edu> Reply-To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org> Date: Thursday, September 21, 2017 at 4:21 PM To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org> Subject: [Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface Hi all, We've encountered an issue with the v2.1.2 Print-to-PDF button in the public interface--apparently for any resource record with an ampersand that is followed immediately by another character that is not a space (e.g. "b" or "AT"), the ampersand is misinterpreted and causes the Print-to-PDF button to fail with an error. For me, that error is just "something went wrong", but the log shows this (when it gets tripped up on "b"): RuntimeError (Failed to clean XML: The reference to entity "w" must end with the ';' delimiter.): So we're guessing ArchivesSpace is thinking "" should be "", and so forth for any other string of text with an ampersand. I checked this by going into a record that wouldn’t print and changing the lone suspect ampersand (“AT” to “AT and T”) and the PDF generated just fine. This doesn't impact being able to just view resource records in the public interface, it's just the PDF function that isn't working. It's a problem, though, because we want to be able to use that PDF functionality but we also have a lot of ampersands in our resource records! Has anyone else experienced this issue or possibly come up with a fix? Thanks, --Benn Benn Joseph Head of Archival Processing Northwestern University Libraries Northwestern University www.library.northwestern.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.library.northwestern.edu=DwMFAg=WO-RGvefibhHBZq3fL85hQ=_Mv1dY22K7jvT5MD7xjbvGVzRDOUMhx4WYcnPSIzYnE=m73cREghXWiIzy9ulXvIZW1Mx-NoJoH_rB1LSdzHQ6Q=Xj5cFVS13R-ioWYCsYqxItOviZziBf6vpg_FBhiC1c4=> benn.jos...@northwestern.edu<mailto:benn.jos...@northwestern.edu%0d> 847.467.6581 ___ Archivesspace_Users_Group mailing list Archivesspace_Users_Group@lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
[Archivesspace_Users_Group] ampersand issue with PDF button in 2.1.2 public interface
Hi all,
We've encountered an issue with the v2.1.2 Print-to-PDF button in the public
interface--apparently for any resource record with an ampersand that is
followed immediately by another character that is not a space (e.g. "b" or
"AT"), the ampersand is misinterpreted and causes the Print-to-PDF button to
fail with an error. For me, that error is just "something went wrong", but the
log shows this (when it gets tripped up on "b"):
RuntimeError (Failed to clean XML: The reference to entity "w" must end with
the ';' delimiter.):
So we're guessing ArchivesSpace is thinking "" should be "", and so forth
for any other string of text with an ampersand. I checked this by going into a
record that wouldn't print and changing the lone suspect ampersand ("AT" to
"AT and T") and the PDF generated just fine.
This doesn't impact being able to just view resource records in the public
interface, it's just the PDF function that isn't working. It's a problem,
though, because we want to be able to use that PDF functionality but we also
have a lot of ampersands in our resource records! Has anyone else experienced
this issue or possibly come up with a fix?
Thanks,
--Benn
Benn Joseph
Head of Archival Processing
Northwestern University Libraries
Northwestern University
www.library.northwestern.edu
benn.jos...@northwestern.edu
847.467.6581
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
