Cant access new docs without restarting Solr (java.nio.file.NoSuchFileException)

[Victor D'agostino]

Hi

I've setup a Solr Cloud 5.5.0 ensemble with ZooKeeper.
If I post a few docs with curl it seems ok :
[root@LXLYOSOL30 ~]# curl --noproxy '*' 
http://lxlyosol30:8983/solr/db/update --data-binary 
@/data/conf-cpm3/test.txt -H 'Content-type:application/xml'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int 
name="QTime">18</int></lst>
</response>


But when I go to the admin page on my first shard I got :
Luke is not configured
although I have in solrconfig.xml and ZooKeeper the line <requestHandler 
name="/admin/luke" 
class="org.apache.solr.handler.admin.LukeRequestHandler" />


If I restart Solr i can see in the stats the new docs have been added !

Statistics :

Last Modified: 3 minutes ago
Num Docs:5
Max Doc:5
Heap Memory Usage:-1

Deleted Docs:0
Version:22
Segment Count:2

Instance :

CWD:/data/solr-5.5.0/server
Instance:/data/solr-5.5.0/server/solr/db_shard1_replica1
Data:/data/solr-5.5.0/server/solr/db_shard1_replica1/data
Index:/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index
Impl:org.apache.solr.core.NRTCachingDirectoryFactory



In the logs I can see a java.nio.file.NoSuchFileException :

149746 INFO  (qtp609396627-17) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/admin/file 
params={file=admin-extra.menu-bottom.html&_=1458667795848&contentType=text/html;charset%3Dutf-8} 
status=404 QTime=1
149780 ERROR (qtp609396627-20) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.h.RequestHandlerBase 
java.nio.file.NoSuchFileException: 
/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index/segments_2
        at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at 
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.Files.size(Files.java:2275)
        at 
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
        at 
org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)
        at 
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)
        at 
org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:137)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
        at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

149780 INFO  (qtp609396627-20) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/admin/luke 
params={numTerms=0&_=1458667795878&show=index&wt=json} status=500 QTime=1
149781 ERROR (qtp609396627-20) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.s.HttpSolrCall 
null:java.nio.file.NoSuchFileException: 
/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index/segments_2
        at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at 
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.Files.size(Files.java:2275)
        at 
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
        at 
org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)
        at 
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)
        at 
org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:137)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)
        at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

149783 INFO  (qtp609396627-17) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/admin/ping 
params={action=status&_=1458667795886&wt=json} status=503 QTime=0
149785 ERROR (qtp609396627-11) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.h.a.ShowFileRequestHandler Can not find: 
/configs/copiemail3/admin-extra.html
149785 INFO  (qtp609396627-11) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/admin/file 
params={file=admin-extra.html&_=1458667795885} status=404 QTime=1
149786 INFO  (qtp609396627-19) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/replication 
params={command=details&_=1458667795882&wt=json} status=0 QTime=4
149793 INFO  (qtp609396627-16) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.S.Request [db_shard1_replica1] 
webapp=/solr path=/admin/system params={_=1458667795884&wt=json} 
status=0 QTime=10
161736 INFO  (commitScheduler-13-thread-1) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start 
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
161763 INFO  (commitScheduler-13-thread-1) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.SolrDeletionPolicy 
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c585e91; 
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
commit{dir=NRTCachingDirectory(MMapDirectory@/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c585e91; 
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
161763 INFO  (commitScheduler-13-thread-1) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit 
generation = 4
161765 INFO  (commitScheduler-13-thread-1) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening 
Searcher@6aca2d82[db_shard1_replica1] realtime
161766 INFO  (commitScheduler-13-thread-1) [c:db s:shard1 r:core_node1 
x:db_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
161826 INFO  (commitScheduler-14-thread-1) [c:db s:shard2 r:core_node2 
x:db_shard2_replica1] o.a.s.u.DirectUpdateHandler2 start 
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
162656 INFO  (commitScheduler-14-thread-1) [c:db s:shard2 r:core_node2 
x:db_shard2_replica1] o.a.s.c.SolrDeletionPolicy 
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/data/solr-5.5.0/server/solr/db_shard2_replica1/data/index 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c585e91; 
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
commit{dir=NRTCachingDirectory(MMapDirectory@/data/solr-5.5.0/server/solr/db_shard2_replica1/data/index 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c585e91; 
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
162656 INFO  (commitScheduler-14-thread-1) [c:db s:shard2 r:core_node2 
x:db_shard2_replica1] o.a.s.c.SolrDeletionPolicy newest commit 
generation = 2
162662 INFO  (commitScheduler-14-thread-1) [c:db s:shard2 r:core_node2 
x:db_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening 
Searcher@55cd91c3[db_shard2_replica1] realtime
162663 INFO  (commitScheduler-14-thread-1) [c:db s:shard2 r:core_node2 
x:db_shard2_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush


Here are my index files :

[root@LXLYOSOL30 ~]# ls -l 
/data/solr-5.5.0/server/solr/db_shard1_replica1/data/index/
total 124
-rw-rw-r-- 1 solr solr  253 22 mars  18:20 _0.fdt
-rw-rw-r-- 1 solr solr   84 22 mars  18:20 _0.fdx
-rw-rw-r-- 1 solr solr 2179 22 mars  18:20 _0.fnm
-rw-rw-r-- 1 solr solr  197 22 mars  18:20 _0_Lucene50_0.doc
-rw-rw-r-- 1 solr solr  113 22 mars  18:20 _0_Lucene50_0.pos
-rw-rw-r-- 1 solr solr 1907 22 mars  18:20 _0_Lucene50_0.tim
-rw-rw-r-- 1 solr solr  603 22 mars  18:20 _0_Lucene50_0.tip
-rw-rw-r-- 1 solr solr   59 22 mars  18:20 _0.nvd
-rw-rw-r-- 1 solr solr   98 22 mars  18:20 _0.nvm
-rw-rw-r-- 1 solr solr  491 22 mars  18:20 _0.si
-rw-rw-r-- 1 solr solr  252 22 mars  18:29 _1.fdt
-rw-rw-r-- 1 solr solr   84 22 mars  18:29 _1.fdx
-rw-rw-r-- 1 solr solr 2179 22 mars  18:29 _1.fnm
-rw-rw-r-- 1 solr solr  197 22 mars  18:29 _1_Lucene50_0.doc
-rw-rw-r-- 1 solr solr  113 22 mars  18:29 _1_Lucene50_0.pos
-rw-rw-r-- 1 solr solr 1907 22 mars  18:29 _1_Lucene50_0.tim
-rw-rw-r-- 1 solr solr  603 22 mars  18:29 _1_Lucene50_0.tip
-rw-rw-r-- 1 solr solr   59 22 mars  18:29 _1.nvd
-rw-rw-r-- 1 solr solr   98 22 mars  18:29 _1.nvm
-rw-rw-r-- 1 solr solr  491 22 mars  18:29 _1.si
-rw-rw-r-- 1 solr solr  244 22 mars  18:30 _2.fdt
-rw-rw-r-- 1 solr solr   84 22 mars  18:30 _2.fdx
-rw-rw-r-- 1 solr solr 2179 22 mars  18:30 _2.fnm
-rw-rw-r-- 1 solr solr  168 22 mars  18:30 _2_Lucene50_0.doc
-rw-rw-r-- 1 solr solr  101 22 mars  18:30 _2_Lucene50_0.pos
-rw-rw-r-- 1 solr solr 1849 22 mars  18:30 _2_Lucene50_0.tim
-rw-rw-r-- 1 solr solr  603 22 mars  18:30 _2_Lucene50_0.tip
-rw-rw-r-- 1 solr solr   59 22 mars  18:30 _2.nvd
-rw-rw-r-- 1 solr solr   98 22 mars  18:30 _2.nvm
-rw-rw-r-- 1 solr solr  491 22 mars  18:30 _2.si
-rw-rw-r-- 1 solr solr  227 22 mars  18:30 segments_4
-rw-rw-r-- 1 solr solr    0 22 mars  18:18 write.lock


Any help ?

Regards
Victor


________________
Ce message et les éventuels documents joints peuvent contenir des informations 
confidentielles. Au cas où il ne vous serait pas destiné, nous vous remercions 
de bien vouloir le supprimer et en aviser immédiatement l'expéditeur. Toute 
utilisation de ce message non conforme à sa destination, toute diffusion ou 
publication, totale ou partielle et quel qu'en soit le moyen est formellement 
interdite. Les communications sur internet n'étant pas sécurisées, l'intégrité 
de ce message n'est pas assurée et la société émettrice ne peut être tenue pour 
responsable de son contenu.