Tim Donohue <mailto:[email protected]>
April 8, 2015 at 7:16 AM
Hi George,
This really has me stumped. It almost starts to sound like a permissions
issue in your index directory (though I know from looking back at this
thread, you said you looked at that).
Since you essentially started with a "fresh" index (no content), and it
still doesn't work, this doesn't sound like an issue with the index
files themselves. While DSpace 5.1 uses a newer version of Solr, it
doesn't act that much different from DSpace 4.3. So, it's confusing to
me that this works for you on DSpace 4.3 and not on DSpace 5.1.
When you encounter that "Index locked for write for core search", there
should be a "write.lock" file sitting in your
/cul/app/dspace/solr/search/data/index/ folder. What are the
permissions on that "write.lock" file? Is it owned by your Tomcat user?
Also is *everything* under /cul/app/dspace/solr/ owned by the Tomcat
user (all files and subdirectories)?
You may even want to try and do the following:
1. Stop Tomcat
2. Recursively change the ownership (just in case), e.g. chown -R
[tomcat-user]:[tomcat-group] /cul/app/dspace/solr/
3. Delete that "write.lock"
4. Start Tomcat again.
5. (You may also need to manually reindex: 'dspace index-discovery -b')
Yea, I know you checked these permissions before, but I'm rather stumped
as to why a fresh, empty index won't even work on your system. I've
never seen that before, and it implies that there may be something lower
level (like a permissions problem) causing problems with Solr.
- Tim
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live
exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette:
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
George Stanley Kozak <mailto:[email protected]>
April 7, 2015 at 12:28 PM
Hi, Tim:
To answer one of your questions: my original DSpace 1.8.2 solr statistics core did have
data in it. Attached is a "snapshot" of the index directory.
Now, I did as you asked. I deleted the all of the solr cores (actually
everything under /cul/app/dspace/solr)
I did the mvn and ant update and everything went well. I copied the webapps
over to tomcat and restarted tomcat. I watched the DSpace log and the
catalina.out log. Everything seemed to move along smoothly. When everything
was done, I waited a few minutes before I went to my DSpace home page. When my
home page came up, I began to see errors in my DSpace log:
ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @
anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.117.195:Error
in Discovery while setting up date facet range:date facet\colon;
dateIssued.year
SolrCore 'search' is not available due to init failure: Index locked for write
for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search'
is not available due to init failure: Index locked for write for core search
Any attempt to do any browse or search results with errors. By the way, the errors on
the web appear with "
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type
application/octet-stream but got text/html." And then raw HTML after that.
What has me stumped is that outside of the problem with the solr statistics,
none of these problems appear in my DSpace 4.3 upgrade. I have no write lock
problems in DSpace 4.3 install.
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
218 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
-----Original Message-----
From: Tim Donohue [mailto:[email protected]]
Sent: Tuesday, April 07, 2015 2:33 PM
To: George Stanley Kozak; [email protected]
Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional
information)
Hi George,
On 4/7/2015 1:09 PM, George Stanley Kozak wrote:
Hi, Tim:
1) I did the "lucene-core-3.5.0.jar" manual update of the indexes. For the
search index, I got no errors, but for the statistics core, I got:
Exception in thread "main" org.apache.lucene.index.IndexNotFoundException:
org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65
at
org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118)
at
org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85)
This would imply that the "search" index upgraded fine (no errors).
But, the error for your "statistics" index almost implies that it's not a valid Solr index. Do you
use the "statistics" index? Is there anything in the
"/cul/app/dspace/solr/statistics/data/index" directory or is it empty?
2) I deleted the statistics core and did the ant update. I brought up my
DSpace 5.1 test system and when I tried to do a search, I got no results. When
I went to do a browse, I got:
ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @
anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.1
17.195:Error in Discovery while setting up date facet range:date
facet\colon; dateIssued.year SolrCore 'search' is not available due to
init failure: Index locked for write for core
search,trace=org.apache.solr.common.SolrException: SolrCore 'search'
is not available due to init failure: Index locked for write for core
search
3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build
and ant update and then copied the new webapps over to tomcat. Everything looked good.
I started tomcat and my site came up. Before doing anything more I ran "./dspace
index-discovery -b". It ran to completion, but all searches return no hits and the
browses give me the same write lock error as above.
I made sure the permissions are correct (dspace user owns the solr indexes and
tomcat) and removed the write lock and tried again, but again I get the same
error.
These other two errors seem rather odd to me...like something else is accessing Solr at
the same time in which you are trying to reindex, and those multiple simultaneous
"writes" are throwing errors.
Can you try something even more simplistic than what you've done above?
1. Stop Tomcat
2. Copy your 1.8.3 data over to where you will install DSpace 5 3. Immediately delete
*everything* under "/cul/app/dspace/solr/"
(including the entire "statistics" and "search" subdirectories). This will wipe
out all existing indexes, obviously.
4. Build DSpace 5.1 (mvn package). Then run "ant update". This should complete,
as you'll have no existing Solr indexes for it to complain about.
5. Finally, start Tomcat back up.
Now, sit back and wait. Don't run any other commands. But, feel free to check
the logs for any issues as Tomcat boots up.
Behind the scenes, DSpace should *automatically* be reindexing your content (and
recreating the "search" index from nothing). You won't need to manually run
anything (like 'index-discovery -b'), as this is all automated in DSpace 5 on the first
bootup. (After the first bootup though, 'index-discovery -b' is the way to force a
reindex.)
I'm just curious if starting from *completely empty* indexes and installing
DSpace 5 (without running 'index-discovery' manually) will act any different.
If you still hit problems here, I'd recommend sending us a full stacktrace of
the error messages you encounter.
- Tim
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
218 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
-----Original Message-----
From: Tim Donohue [mailto:[email protected]]
Sent: Tuesday, April 07, 2015 11:52 AM
To: George Stanley Kozak; [email protected]
Subject: Re: [Dspace-tech] Error after install of DSpace 5.1
(Additional information)
Hi George,
What happens if you do the following against your old 1.8.2 indexes:
1. wget
"http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar";
2. java -cp lucene-core-3.5.0.jar
org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/statistics/data/index
3. java -cp lucene-core-3.5.0.jar
org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/search/data/index
Do either of those "IndexUpgrader" commands succeed against your *old*
1.8.2 indexes? If not, could you send along the exception stacktrace?
I'm just trying to determine if the issue is actually in these old 1.8.2
indexes. It may be either they are indexes from an unexpected, older version of
Solr, or there's some minor index corruption here.
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Tim Donohue <mailto:[email protected]>
April 7, 2015 at 11:33 AM
Hi George,
On 4/7/2015 1:09 PM, George Stanley Kozak wrote:
Hi, Tim:
1) I did the "lucene-core-3.5.0.jar" manual update of the indexes. For the
search index, I got no errors, but for the statistics core, I got:
Exception in thread "main" org.apache.lucene.index.IndexNotFoundException:
org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65
at
org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118)
at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85)
This would imply that the "search" index upgraded fine (no errors).
But, the error for your "statistics" index almost implies that it's not
a valid Solr index. Do you use the "statistics" index? Is there
anything in the "/cul/app/dspace/solr/statistics/data/index" directory
or is it empty?
2) I deleted the statistics core and did the ant update. I brought up my
DSpace 5.1 test system and when I tried to do a search, I got no results. When
I went to do a browse, I got:
ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @
anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.117.195:Error
in Discovery while setting up date facet range:date facet\colon;
dateIssued.year
SolrCore 'search' is not available due to init failure: Index locked for write
for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search'
is not available due to init failure: Index locked for write for core search
3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build
and ant update and then copied the new webapps over to tomcat. Everything looked good.
I started tomcat and my site came up. Before doing anything more I ran "./dspace
index-discovery -b". It ran to completion, but all searches return no hits and the
browses give me the same write lock error as above.
I made sure the permissions are correct (dspace user owns the solr indexes and
tomcat) and removed the write lock and tried again, but again I get the same
error.
These other two errors seem rather odd to me...like something else is
accessing Solr at the same time in which you are trying to reindex, and
those multiple simultaneous "writes" are throwing errors.
Can you try something even more simplistic than what you've done above?
1. Stop Tomcat
2. Copy your 1.8.3 data over to where you will install DSpace 5
3. Immediately delete *everything* under "/cul/app/dspace/solr/"
(including the entire "statistics" and "search" subdirectories). This
will wipe out all existing indexes, obviously.
4. Build DSpace 5.1 (mvn package). Then run "ant update". This should
complete, as you'll have no existing Solr indexes for it to complain about.
5. Finally, start Tomcat back up.
Now, sit back and wait. Don't run any other commands. But, feel free to
check the logs for any issues as Tomcat boots up.
Behind the scenes, DSpace should *automatically* be reindexing your
content (and recreating the "search" index from nothing). You won't need
to manually run anything (like 'index-discovery -b'), as this is all
automated in DSpace 5 on the first bootup. (After the first bootup
though, 'index-discovery -b' is the way to force a reindex.)
I'm just curious if starting from *completely empty* indexes and
installing DSpace 5 (without running 'index-discovery' manually) will
act any different. If you still hit problems here, I'd recommend sending
us a full stacktrace of the error messages you encounter.
- Tim
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
218 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
-----Original Message-----
From: Tim Donohue [mailto:[email protected]]
Sent: Tuesday, April 07, 2015 11:52 AM
To: George Stanley Kozak; [email protected]
Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional
information)
Hi George,
What happens if you do the following against your old 1.8.2 indexes:
1. wget
"http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar";
2. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/statistics/data/index
3. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/search/data/index
Do either of those "IndexUpgrader" commands succeed against your *old*
1.8.2 indexes? If not, could you send along the exception stacktrace?
I'm just trying to determine if the issue is actually in these old 1.8.2
indexes. It may be either they are indexes from an unexpected, older version of
Solr, or there's some minor index corruption here.
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
George Stanley Kozak <mailto:[email protected]>
April 7, 2015 at 11:09 AM
Hi, Tim:
1) I did the "lucene-core-3.5.0.jar" manual update of the indexes. For
the search index, I got no errors, but for the statistics core, I got:
Exception in thread "main"
org.apache.lucene.index.IndexNotFoundException:
org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65
at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118)
at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85)
2) I deleted the statistics core and did the ant update. I brought up
my DSpace 5.1 test system and when I tried to do a search, I got no
results. When I went to do a browse, I got:
ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @
anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.117.195:Error
in Discovery while setting up date facet range:date facet\colon;
dateIssued.year
SolrCore 'search' is not available due to init failure: Index locked
for write for core search,trace=org.apache.solr.common.SolrException:
SolrCore 'search' is not available due to init failure: Index locked
for write for core search
3) I stopped tomcat, deleted the search index (as you suggested) and
did the mvn build and ant update and then copied the new webapps over
to tomcat. Everything looked good. I started tomcat and my site came
up. Before doing anything more I ran "./dspace index-discovery -b". It
ran to completion, but all searches return no hits and the browses
give me the same write lock error as above.
I made sure the permissions are correct (dspace user owns the solr
indexes and tomcat) and removed the write lock and tried again, but
again I get the same error.
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
218 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
-----Original Message-----
From: Tim Donohue [mailto:[email protected]]
Sent: Tuesday, April 07, 2015 11:52 AM
To: George Stanley Kozak; [email protected]
Subject: Re: [Dspace-tech] Error after install of DSpace 5.1
(Additional information)
Hi George,
What happens if you do the following against your old 1.8.2 indexes:
1. wget
"http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar";
2. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/statistics/data/index
3. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader
/cul/app/dspace/solr/search/data/index
Do either of those "IndexUpgrader" commands succeed against your *old*
1.8.2 indexes? If not, could you send along the exception stacktrace?
I'm just trying to determine if the issue is actually in these old
1.8.2 indexes. It may be either they are indexes from an unexpected,
older version of Solr, or there's some minor index corruption here.
George Stanley Kozak <mailto:[email protected]>
April 7, 2015 at 8:12 AM
Thanks for your help, Tim:
I'm afraid that I keep hitting a wall when I try to upgrade to DSpace 5.1
I tried to go directly from DSpace 1.8.2 to DSpace 5.1 and got:
ERROR occurred while checking Solr index version:
Exception in thread "main" java.io.IOException: Could not read Lucene
segments files in /cul/app/dspace/solr/statistics/data/index
We have been using our own stats gathering system instead of the one
within DSpace. So, I removed the statistics index but then I got:
Exception in thread "main" java.io.IOException: Could not read Lucene
segments files in /cul/app/dspace/solr/search/data/index
This version of Lucene only supports indexes created with release 3.0
and later.
So, I went ahead and used the DSpace 4.3 version of our solr indexes
(without the statistics). It ran fine and everything got upgraded, but
when I tried to do any searching, I get the "write lock error". If I
turn off tomcat, delete the write lock and restart things, I get the
same thing. Also, I cannot update the discovery index because of the
write lock problem.
By the way, I did try the manual upgrade of the Solr indexes defined
in the manual, but that didn't work for me, either.
So, right now, it looks like I can only upgrade to DSpace 4.3, unless
you have some more suggestions for me.
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
218 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
-----Original Message-----
From: Tim Donohue [mailto:[email protected]]
Sent: Tuesday, April 07, 2015 9:31 AM
To: George Stanley Kozak; [email protected]
Subject: Re: [Dspace-tech] Error after install of DSpace 5.1
(Additional information)
Hi George,
Yes, that is what I was suggesting...upgrading directly from 1.8.2 or
3.3 (whichever was your starting point) to 5.1. The "ant update"
should take care of the necessary Solr index upgrades for you.
- Tim
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live
exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette:
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
