I was able to replicate this problem using Rice's data. I set up a fresh instance running 1.4.2 using their database dump from that version, and it built the index fine (took 10-15 minutes). When I migrated it to 1.5, the indexer seems to get hung up after indexing the top containers (which finished about 2 hrs ago). Below is an error message that appears right after it finishes the top containers (and continues to appear repeatedly in the log file). It seems to be related to indexing archival objects, though I can't say I understand anything more than that:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ Indexed 12884 top_container records in 600 seconds ~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sep 15, 2016 9:33:57 AM org.eclipse.jetty.server.handler.ContextHandler$Context log INFO: DEBUG: resetting rack response due exception Failure in periodic indexer worker thread: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd";> ... <body> <div id="summary"> <h1>Internal Server Error <span>(500)</span></h1> <table class="meta"> <tr> <th>Request Method:</th> <td>GET</td> </tr> <tr> <th>Request URL:</th> <td>http://localhost:8089/repositories/2/archival_objects?id_set=1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16%2C17%2C18%2C19%2C20%2C21%2C22%2C23%2C24%2C25&resolve%5B%5D=location_profile&resolve%5B%5D=container_profile&resolve%5B%5D=container_locations&resolve%5B%5D=subjects&resolve%5B%5D=linked_agents&resolve%5B%5D=linked_records&resolve%5B%5D=classifications&resolve%5B%5D=digital_object</td> </tr> </table> </div> <div id="info"> <p>(NoMethodError) undefined method `related_records' for nil:NilClass</p> </div> <div id="explanation"> <p> You're seeing this error because you use <code>JRuby::Rack::ErrorApp::ShowStatus</code>. </p> </div> </body> ------------------------------- Rachel Maderik Systems and Technology Librarian 501D Preston Library Virginia Military Institute Lexington, VA 24450 540-464-7572 From: [email protected] [mailto:[email protected]] On Behalf Of Mang Sun Sent: Wednesday, September 14, 2016 3:00 PM To: Archivesspace Users Group <[email protected]> Subject: Re: [Archivesspace_Users_Group] hELP - No record found after test migration from v1.4.2 to 1.5.0 Christine and other AS community members who have successfully migrated AS 1.4.2 to AS 1.5.x, We are trying to draw attention and seek help from AS development team and other community members who have successfully migrated AS 1.4.2 to AS 1.5.x. If necessary, we are willing to provide our MySQL dump file for you to test. Test migration from AS 1.4.2 to AS1.5.x really drives us mad. Though we don't quite see any (out of mem) error in the log, we still blindly pump up heap size (Xmx )to 6GB, stack size set o 3MB and rerun rebuild index or the complete migration steps, the problem is still there - Solr seemingly very silently stops writing or committing index files when the number of docs in Solr reaches to about 15,xxx as opposed to 500,xxx indexed docs found in our 1.4.2 instance, though thereafter we still can see the " end_commit_flush" message pop up in the log from time to time .Our test box environments are 1. Redhat 6.8 ,JAVA build 1.7.0_67 . MySQL 5.1.7 and 8GB memory 2.. Redhat 7.2 Virtual Machine,NYSQL 5.5.45, java 1.80_65,and 4GB. We also can confirm database side migration is largely done with new tables and fields added, existing records converted and top container records added. The problem is sticking to rebuilding the whole index - Solr stops writing and committing index prematurely after the number of docs grows to a small number(12xxx-15xxx) though commit flush can still be seen thereafter . Now we have exhausted ideas and means in figuring out why this "no record found " problem that seemingly only happens to us. Other experiment we did is export EAD file of one resource record from our production and load it into a 1.4.2 test instance and then upgrade this test instance to 1.5.x. this time the resource record can be indexed and appear(some issue with the Name ) in the test 1.5.x instance. I think there is some fundamental but very sneaky issue with 1.5.x indexing (rebuild index). The challenge is that no explicit error message that could be easily found in the log that can give us some hint and therefore this is a nightmare given the long waiting time before you can confirm a successful indexing or not. May be a standalone tool allowing user to manually rebuild index could be very helpful and straightforward, and less arcane. As mentioned earlier, records have been converted and migrated but not indexed, what else we tried is use direct route path (can be inferred from our production) to pull records out and update them by toggling through Suppress/Unsuppress or Publisesh/Unpublished , then we can get those updated records indexed. in the following diagram, we can see 3 resources, 2 agent records and some other record types highlighted in yellow appear in the types field, as opposed to the types (not highlighted) originally indexed and generated by the migration process. Also in the same diagram, the number of indexed docs is too small (we can see about half million docs in our 1.4.2 production instance).Also we found a side issue with agent record, no matter what you set for Publish, Solr only sees it unpublished.But At least, this experiment can confirm there is not obvious problem with dynamic indexing. If AS developers or other AS users want to reproduce the issue or give a try for a solution, we can provide our v1.4.2 database dump or the v1.5.x database dump. Thank you very much. Mang Sun Rice U. [cid:[email protected]] On 9/9/2016 11:55 AM, Christine Di Bella wrote: Mang, How long ago did you start the reindex and how big is your database? The v1.5.1 upgrade improved indexing speed, especially for large databases, but the indexing does still take some time. People on this list can probably give you a sense of how long it took them for different sizes of databases. It's probably been most common to run it overnight or over a weekend, when use of the system is low. (Some people were large databases were experiencing problems with indexing taking an excessive amount of time or not completing at all, which was one of the reasons for the quick release of v1.5.1.) Christine Christine Di Bella Community Outreach Manager [email protected]<mailto:[email protected]> 800.999.8558 x2905 678-235-2905 cdibella13 (Skype) [cid:[email protected]] From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Mang Sun Sent: Friday, September 9, 2016 12:31 PM To: Archivesspace Users Group <[email protected]><mailto:[email protected]> Subject: Re: [Archivesspace_Users_Group] No record found after test migration from v1.4.2 to 1.5.0 I even didn't copy the data folder of the v1.4.2 into v.1.5.0. but even after I follow the recommended way by copying the data folder and removing those two subfolders mentioned by Joshua the problem is still there.Peeping into Solr schema, I found field "types" only have the following four types but not resource and digital_object,subject and agent . The missing types may explain why we find no records under every record type tab. Puzzled~ Mang * 12608 o top_container<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22top_container%22> * 1,169 o tree_view<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22tree_view%22> * 75 o archival_object<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22archival_object%22> * 1 o repository<http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22repository%22> On 9/9/2016 8:52 AM, Sally Vermaaten wrote: Hi Mang Sun, I agree with Joshua - this sounds to me like an indexing problem to me. In the course of upgrading to 1.5.0 did you delete the data/solr_index/index directory and all files in the data/indexer_state directory? This is one of the recommended steps in the instructions for upgrading to v.1.5.0<https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md>. We accidentally missed this step once in a running a test upgrade and definitely experienced some indexing problems though I don't remember the details. As it was a test upgrade (and it sounds like that's what you're doing as well) we just started afresh and ran the upgrade process over again. When we included that step, things were fine. On a related note - there's a 1.5.1 release of ArchivesSpace<https://github.com/archivesspace/archivesspace/releases/tag/v1.5.1> that includes several fixes for bugs that were found in the weeks after 1.5.0 was released. If are looking to update to a 1.5.x version, you may want to consider upgrading to 1.5.1 one rather than 1.5.0. Good luck, Sally On Fri, Sep 9, 2016 at 9:50 AM, Joshua D. Shaw <[email protected]<mailto:[email protected]>> wrote: I believe the suggestion for migration to 1.5+ was to delete the existing index and indexer states which will trigger a complete reindex. Best, Joshua On Sep 9, 2016, at 8:20 AM, "Mang Sun" <[email protected]<mailto:[email protected]>> wrote: > i > > After we migrate our v.1.4.2 test instance to v1.5.0, we can't find any > records but just no record in Staff interface and public interface. > > But if we open a migrated record (saycollection record) using the direct > route path to that record and resave it, this record but only this one starts > to show up under Collections when browsing collections. This problem does not > come with any new record created within v.1.5.0. Any suggestion? > > Thank you. > > Mang Sun > > Rice U. > > > > _______________________________________________ > Archivesspace_Users_Group mailing list > [email protected]<mailto:[email protected]> > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list [email protected]<mailto:[email protected]> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -- Sally Vermaaten Project Manager, Archival Systems New York University Libraries 1-212-992-6259 _______________________________________________ Archivesspace_Users_Group mailing list [email protected]<mailto:[email protected]> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list [email protected]<mailto:[email protected]> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
_______________________________________________ Archivesspace_Users_Group mailing list [email protected] http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
