Hi all，
I found an exception when running Solr on HDFS。The detail is：
Running solr on HDFS，and update doc was running always，
then，kill -9 solr JVM or reboot linux os/shutdown linux os，then restart all.
The exception appears like：
2018-08-26 22:23:12.529 ERROR
(coreContainerWorkExecutor-2-thread-1-processing-n:cluster-node001:8983_solr)
[ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on
startup
org.apache.solr.common.SolrException: Unable to create core
[collection002_shard56_replica_n110]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1061)
at
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:640)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.solr.common.SolrException: Index dir
'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core
'collection002_shard56_replica_n110' is already locked. The most likely
cause is another Solr server (or another solr core in this server) also
configured to use this directory; other possible causes may be specific to
lockType: hdfs
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1009)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:864)
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1040)
... 7 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir
'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core
'collection002_shard56_replica_n110' is already locked. The most likely
cause is another Solr server (or another solr core in this server) also
configured to use this directory; other possible causes may be specific to
lockType: hdfs
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:746)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:955)
... 9 more
In fact, a print out a hdfs api level exception stack, it reports like:
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
/solr/collection002/core_node17/data/index/write.lock for client
192.168.0.12 already exists
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2563)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2450)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2334)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at sun.reflect.GeneratedConstructorAccessor140.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1839)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at
org.apache.solr.store.hdfs.HdfsLockFactory.obtainLock(HdfsLockFactory.java:68)
So my question is , why a solr server can not recognize the write.lock file
is created by itself before?
Is there any solution to solve this kind of accident or failure？
Thanks~
TinsWzy
