Thanks for your answer! @Erick Erickson <erickerick...@gmail.com> So, It's not recommended to run Solr on NFS ( like HDFS) now? Maybe because of crash error or performance problem. I have a look at SOLR-8335&SOLR-8169, there is no good solution for this now， And maybe manual removal is the best option?
Erick Erickson <erickerick...@gmail.com> 于2018年8月27日周一 上午11:41写道： > Because HDFS doesn't follow the file semantics that Solr expects. > > There's quite a bit of background here: > https://issues.apache.org/jira/browse/SOLR-8335 > > Best, > Erick > On Sun, Aug 26, 2018 at 6:47 PM zhenyuan wei <tins...@gmail.com> wrote: > > > > Hi all， > > I found an exception when running Solr on HDFS。The detail is： > > Running solr on HDFS，and update doc was running always， > > then，kill -9 solr JVM or reboot linux os/shutdown linux os，then restart > all. > > The exception appears like： > > > > 2018-08-26 22:23:12.529 ERROR > > > (coreContainerWorkExecutor-2-thread-1-processing-n:cluster-node001:8983_solr) > > [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on > > startup > > org.apache.solr.common.SolrException: Unable to create core > > [collection002_shard56_replica_n110] > > at > > > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1061) > > at > > org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:640) > > at > > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > > at java.lang.Thread.run(Thread.java:834) > > Caused by: org.apache.solr.common.SolrException: Index dir > > 'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core > > 'collection002_shard56_replica_n110' is already locked. The most likely > > cause is another Solr server (or another solr core in this server) also > > configured to use this directory; other possible causes may be specific > to > > lockType: hdfs > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1009) > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:864) > > at > > > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1040) > > ... 7 more > > Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir > > 'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core > > 'collection002_shard56_replica_n110' is already locked. The most likely > > cause is another Solr server (or another solr core in this server) also > > configured to use this directory; other possible causes may be specific > to > > lockType: hdfs > > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:746) > > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:955) > > ... 9 more > > > > > > In fact, a print out a hdfs api level exception stack, it reports like: > > > > Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: > > /solr/collection002/core_node17/data/index/write.lock for client > > 192.168.0.12 already exists > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2563) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2450) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2334) > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623) > > at > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397) > > at > > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045) > > > > at > sun.reflect.GeneratedConstructorAccessor140.newInstance(Unknown > > Source) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > > > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > > at > > > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > > at > > > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1839) > > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689) > > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) > > at > > > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > > at > > > org.apache.solr.store.hdfs.HdfsLockFactory.obtainLock(HdfsLockFactory.java:68) > > > > So my question is , why a solr server can not recognize the write.lock > file > > is created by itself before? > > Is there any solution to solve this kind of accident or failure？ > > > > > > > > Thanks~ > > TinsWzy >
