Because HDFS doesn't follow the file semantics that Solr expects. There's quite a bit of background here: https://issues.apache.org/jira/browse/SOLR-8335
Best, Erick On Sun, Aug 26, 2018 at 6:47 PM zhenyuan wei <tins...@gmail.com> wrote: > > Hi all， > I found an exception when running Solr on HDFS。The detail is： > Running solr on HDFS，and update doc was running always， > then，kill -9 solr JVM or reboot linux os/shutdown linux os，then restart all. > The exception appears like： > > 2018-08-26 22:23:12.529 ERROR > (coreContainerWorkExecutor-2-thread-1-processing-n:cluster-node001:8983_solr) > [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on > startup > org.apache.solr.common.SolrException: Unable to create core > [collection002_shard56_replica_n110] > at > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1061) > at > org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:640) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:834) > Caused by: org.apache.solr.common.SolrException: Index dir > 'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core > 'collection002_shard56_replica_n110' is already locked. The most likely > cause is another Solr server (or another solr core in this server) also > configured to use this directory; other possible causes may be specific to > lockType: hdfs > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1009) > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:864) > at > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1040) > ... 7 more > Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir > 'hdfs://hdfs-cluster/solr/collection002/core_node113/data/index/' of core > 'collection002_shard56_replica_n110' is already locked. The most likely > cause is another Solr server (or another solr core in this server) also > configured to use this directory; other possible causes may be specific to > lockType: hdfs > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:746) > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:955) > ... 9 more > > > In fact, a print out a hdfs api level exception stack, it reports like: > > Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: > /solr/collection002/core_node17/data/index/write.lock for client > 192.168.0.12 already exists > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2563) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2450) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2334) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045) > > at sun.reflect.GeneratedConstructorAccessor140.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1839) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) > at > org.apache.solr.store.hdfs.HdfsLockFactory.obtainLock(HdfsLockFactory.java:68) > > So my question is , why a solr server can not recognize the write.lock file > is created by itself before? > Is there any solution to solve this kind of accident or failure？ > > > > Thanks~ > TinsWzy
