Hi, Tim: Yes, this has me stumped. Here is the permissions on the writelock:
-rw-rw-r-- 1 dspace dspace 0 Apr 8 09:59 write.lock Tomcat is owned by the dspace user: drwxr-xr-x 3 dspace dspace 4096 Apr 2 2014 tomcat Solr is owned by the dspace user: drwxrwxr-x 6 dspace dspace 4096 Apr 8 09:58 solr I did as you suggested and stopped tomcat, recursively changed the permissions on the solr indexes: sudo chown -R dspace:dspace /cul/app/dspace/solr I deleted the write.lock and restarted tomcat. I then did /cul/app/dspace/bin/dspace index-discovery -b Immediately, I received the write lock error and the re-index did not work. I restored the postgres database and solr indexes to what they were before and did a mvn build and ant update of my DSpace 4.3 source and everything works fine(?!?!) There is obviously some subtle difference on how DSpace 5.1 and DSpace 4.3 (and lower) are handling the solr indexes and my server set up is just right (or wrong) enough to cause this problem. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -----Original Message----- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Wednesday, April 08, 2015 10:16 AM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, This really has me stumped. It almost starts to sound like a permissions issue in your index directory (though I know from looking back at this thread, you said you looked at that). Since you essentially started with a "fresh" index (no content), and it still doesn't work, this doesn't sound like an issue with the index files themselves. While DSpace 5.1 uses a newer version of Solr, it doesn't act that much different from DSpace 4.3. So, it's confusing to me that this works for you on DSpace 4.3 and not on DSpace 5.1. When you encounter that "Index locked for write for core search", there should be a "write.lock" file sitting in your /cul/app/dspace/solr/search/data/index/ folder. What are the permissions on that "write.lock" file? Is it owned by your Tomcat user? Also is *everything* under /cul/app/dspace/solr/ owned by the Tomcat user (all files and subdirectories)? You may even want to try and do the following: 1. Stop Tomcat 2. Recursively change the ownership (just in case), e.g. chown -R [tomcat-user]:[tomcat-group] /cul/app/dspace/solr/ 3. Delete that "write.lock" 4. Start Tomcat again. 5. (You may also need to manually reindex: 'dspace index-discovery -b') Yea, I know you checked these permissions before, but I'm rather stumped as to why a fresh, empty index won't even work on your system. I've never seen that before, and it implies that there may be something lower level (like a permissions problem) causing problems with Solr. - Tim On 4/7/2015 2:28 PM, George Stanley Kozak wrote: > Hi, Tim: > > To answer one of your questions: my original DSpace 1.8.2 solr statistics > core did have data in it. Attached is a "snapshot" of the index directory. > > Now, I did as you asked. I deleted the all of the solr cores > (actually everything under /cul/app/dspace/solr) > > I did the mvn and ant update and everything went well. I copied the webapps > over to tomcat and restarted tomcat. I watched the DSpace log and the > catalina.out log. Everything seemed to move along smoothly. When everything > was done, I waited a few minutes before I went to my DSpace home page. When > my home page came up, I began to see errors in my DSpace log: > ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ > anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.1 > 17.195:Error in Discovery while setting up date facet range:date > facet\colon; dateIssued.year SolrCore 'search' is not available due to > init failure: Index locked for write for core > search,trace=org.apache.solr.common.SolrException: SolrCore 'search' > is not available due to init failure: Index locked for write for core > search > > Any attempt to do any browse or search results with errors. By the way, the > errors on the web appear with " > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > Expected mime type application/octet-stream but got text/html." And then raw > HTML after that. > > What has me stumped is that outside of the problem with the solr statistics, > none of these problems appear in my DSpace 4.3 upgrade. I have no write lock > problems in DSpace 4.3 install. > > George Kozak > Digital Library Specialist > Cornell University Library Information Technologies (CUL-IT) > 218 Olin Library > Cornell University > Ithaca, NY 14853 > 607-255-8924 > > > > > -----Original Message----- > From: Tim Donohue [mailto:tdono...@duraspace.org] > Sent: Tuesday, April 07, 2015 2:33 PM > To: George Stanley Kozak; dspace-tech@lists.sourceforge.net > Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 > (Additional information) > > > Hi George, > > On 4/7/2015 1:09 PM, George Stanley Kozak wrote: >> Hi, Tim: >> >> 1) I did the "lucene-core-3.5.0.jar" manual update of the indexes. For the >> search index, I got no errors, but for the statistics core, I got: >> Exception in thread "main" org.apache.lucene.index.IndexNotFoundException: >> org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 >> at >> org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) >> at >> org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) > > This would imply that the "search" index upgraded fine (no errors). > > But, the error for your "statistics" index almost implies that it's not a > valid Solr index. Do you use the "statistics" index? Is there anything in > the "/cul/app/dspace/solr/statistics/data/index" directory or is it empty? > > >> 2) I deleted the statistics core and did the ant update. I brought up my >> DSpace 5.1 test system and when I tried to do a search, I got no results. >> When I went to do a browse, I got: >> ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer >> @ >> anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84. >> 1 17.195:Error in Discovery while setting up date facet range:date >> facet\colon; dateIssued.year SolrCore 'search' is not available due >> to init failure: Index locked for write for core >> search,trace=org.apache.solr.common.SolrException: SolrCore 'search' >> is not available due to init failure: Index locked for write for core >> search >> >> 3) I stopped tomcat, deleted the search index (as you suggested) and did >> the mvn build and ant update and then copied the new webapps over to tomcat. >> Everything looked good. I started tomcat and my site came up. Before >> doing anything more I ran "./dspace index-discovery -b". It ran to >> completion, but all searches return no hits and the browses give me the same >> write lock error as above. >> I made sure the permissions are correct (dspace user owns the solr indexes >> and tomcat) and removed the write lock and tried again, but again I get the >> same error. > > These other two errors seem rather odd to me...like something else is > accessing Solr at the same time in which you are trying to reindex, and those > multiple simultaneous "writes" are throwing errors. > > Can you try something even more simplistic than what you've done above? > > 1. Stop Tomcat > 2. Copy your 1.8.3 data over to where you will install DSpace 5 3. > Immediately delete *everything* under "/cul/app/dspace/solr/" > (including the entire "statistics" and "search" subdirectories). This will > wipe out all existing indexes, obviously. > 4. Build DSpace 5.1 (mvn package). Then run "ant update". This should > complete, as you'll have no existing Solr indexes for it to complain about. > 5. Finally, start Tomcat back up. > > Now, sit back and wait. Don't run any other commands. But, feel free to check > the logs for any issues as Tomcat boots up. > > Behind the scenes, DSpace should *automatically* be reindexing your > content (and recreating the "search" index from nothing). You won't > need to manually run anything (like 'index-discovery -b'), as this is > all automated in DSpace 5 on the first bootup. (After the first bootup > though, 'index-discovery -b' is the way to force a reindex.) > > I'm just curious if starting from *completely empty* indexes and installing > DSpace 5 (without running 'index-discovery' manually) will act any different. > If you still hit problems here, I'd recommend sending us a full stacktrace of > the error messages you encounter. > > - Tim > > > >> >> George Kozak >> Digital Library Specialist >> Cornell University Library Information Technologies (CUL-IT) >> 218 Olin Library >> Cornell University >> Ithaca, NY 14853 >> 607-255-8924 >> >> >> -----Original Message----- >> From: Tim Donohue [mailto:tdono...@duraspace.org] >> Sent: Tuesday, April 07, 2015 11:52 AM >> To: George Stanley Kozak; dspace-tech@lists.sourceforge.net >> Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 >> (Additional information) >> >> Hi George, >> >> What happens if you do the following against your old 1.8.2 indexes: >> >> 1. wget >> "http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar"; >> >> 2. java -cp lucene-core-3.5.0.jar >> org.apache.lucene.index.IndexUpgrader >> /cul/app/dspace/solr/statistics/data/index >> >> 3. java -cp lucene-core-3.5.0.jar >> org.apache.lucene.index.IndexUpgrader >> /cul/app/dspace/solr/search/data/index >> >> Do either of those "IndexUpgrader" commands succeed against your >> *old* >> 1.8.2 indexes? If not, could you send along the exception stacktrace? >> >> I'm just trying to determine if the issue is actually in these old 1.8.2 >> indexes. It may be either they are indexes from an unexpected, older version >> of Solr, or there's some minor index corruption here. >> ------------------------------------------------------------------------------ BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF _______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette ------------------------------------------------------------------------------ BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF _______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
