Re: unable to load core after cluster restart
Hi, So finally we got our jdk/jre upgraded to 1.6.0-33. but it didnt solve the problem. I am still seeing same write.lock error. I was able to solve the problem by changing the lock type from native to single. but I am not sure about other ramifications of this approach. Do you see any problems coming into my way with 1 shard/4 replication setup. Thanks, Kaustubh -- View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4100538.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Hi All, I have further investigated the difference between both the environments. We have JDK 1.6.0_17 (VM 14.3-b01)on UAT and JDK 1.6.0_33 (VM 20.8-b03)on QA1. Can it be the reason behind this error? Is there a recommended jdk version for SolrCloud ? Thanks, Kaustubh -- View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099621.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
On 11/6/2013 11:53 AM, kaustubh147 wrote: Hi All, I have further investigated the difference between both the environments. We have JDK 1.6.0_17 (VM 14.3-b01)on UAT and JDK 1.6.0_33 (VM 20.8-b03)on QA1. Can it be the reason behind this error? Is there a recommended jdk version for SolrCloud ? For Solr 4.x, Java 6 is required, and Java 7 works just fine. Java 6 is no longer receiving unpaid support from Oracle, and there will likely be no more public versions available, which makes Java 7 attractive. You don't need the JDK, the JRE is sufficient, but having the JDK is not a problem. Oracle's product is recommended for Java 6. For Java 7, OpenJDK is probably sufficient, because for that version OpenJDK is the reference implementation. Naturally, the latest version is always recommended, at least for Java 6. In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems with Lucene/Solr. For Java 7, 1.7.0_25 is currently the recommended version, and hopefully Oracle will be fixing their bug in the next release. I don't know what might be causing the problems you're seeing when following my advice. You'd have to give a lot of config details and the errors you're seeing. Thanks, Shawn
Re: unable to load core after cluster restart
Hi,
Here is my
solr.xml
solr
solrcloud
str name=host${host:}/str
int name=hostPort28081/int
str name=hostContext/solr/str
str name=zkHostIP1:2181,IP2:2181,IP3:2181/mysolr/str
int name=zkClientTimeout15000/int
bool name=genericCoreNodeNames${genericCoreNodeNames:true}/bool
/solrcloud
shardHandlerFactory name=shardHandlerFactory
class=HttpShardHandlerFactory
int name=socketTimeout${socketTimeout:0}/int
int name=connTimeout${connTimeout:0}/int
/shardHandlerFactory
/solr
--
and
solrconfig.xml
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersion4.5/luceneMatchVersion
lib dir=../../../contrib/extraction/lib regex=.*\.jar /
lib dir=../../../dist/ regex=solr-cell-\d.*\.jar /
lib dir=../../../contrib/clustering/lib/ regex=.*\.jar /
lib dir=../../../dist/ regex=solr-clustering-\d.*\.jar /
lib dir=../../../dist/ regex=solr-dataimporthandler-\d.*\.jar /
lib dir=../../../contrib/dataimporthandler/lib/ regex=.*\.jar /
lib dir=../../../contrib/langid/lib/ regex=.*\.jar /
lib dir=../../../dist/ regex=solr-langid-\d.*\.jar /
lib dir=../../../contrib/velocity/lib regex=.*\.jar /
lib dir=../../../dist/ regex=solr-velocity-\d.*\.jar /
dataDir${solr.data.dir:}/dataDir
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
codecFactory class=solr.SchemaCodecFactory/
schemaFactory class=ClassicIndexSchemaFactory/
indexConfig
writeLockTimeout2/writeLockTimeout
lockType${solr.lock.type:native}/lockType
infoStreamtrue/infoStream
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
updateLog
str name=dir${solr.data.dir:}/str
/updateLog
autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
/autoSoftCommit
/updateHandler
query
maxBooleanClauses50/maxBooleanClauses
filterCache class=solr.FastLRUCache
size=512
initialSize=512
autowarmCount=0/
queryResultCache class=solr.LRUCache
size=512
initialSize=512
autowarmCount=0/
documentCache class=solr.LRUCache
size=512
initialSize=512
autowarmCount=0/
cache name=perSegFilter
class=solr.search.LRUCache
size=10
initialSize=0
autowarmCount=10
regenerator=solr.NoOpRegenerator /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
queryResultWindowSize20/queryResultWindowSize
queryResultMaxDocsCached200/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
/arr
/listener
listener event=firstSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=qstatic firstSearcher warming in solrconfig.xml/str
/lst
/arr
/listener
useColdSearcherfalse/useColdSearcher
maxWarmingSearchers2/maxWarmingSearchers
/query
requestDispatcher handleSelect=false
requestParsers enableRemoteStreaming=true
multipartUploadLimitInKB=2048000
formdataUploadLimitInKB=16384
addHttpRequestToContext=false/
httpCaching never304=true /
/requestDispatcher
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=configdb-data-config.xml/str
/lst
/requestHandler
requestHandler name=/select class=solr.SearchHandler
lst name=defaults
str name=echoParamsexplicit/str
int name=rows10/int
str name=dfkeyword/str
/lst
/requestHandler
requestHandler name=/query class=solr.SearchHandler
lst name=defaults
str name=echoParamsexplicit/str
str name=wtjson/str
str name=indenttrue/str
str name=dfkeyword/str
/lst
/requestHandler
requestHandler name=/get class=solr.RealTimeGetHandler
lst name=defaults
str name=omitHeadertrue/str
str name=wtjson/str
str name=indenttrue/str
/lst
/requestHandler
requestHandler name=/update class=solr.UpdateRequestHandler
/requestHandler
requestHandler name=/update/json class=solr.JsonUpdateRequestHandler
lst name=defaults
str name=stream.contentTypeapplication/json/str
/lst
/requestHandler
requestHandler name=/update/csv class=solr.CSVRequestHandler
lst name=defaults
str name=stream.contentTypeapplication/csv/str
/lst
/requestHandler
requestHandler name=/update/extract
startup=lazy
class=solr.extraction.ExtractingRequestHandler
lst name=defaults
str name=lowernamestrue/str
str name=uprefixignored_/str
str name=captureAttrtrue/str
str name=fmap.alinks/str
str name=fmap.divignored_/str
/lst
/requestHandler
requestHandler name=/analysis/field
startup=lazy
class=solr.FieldAnalysisRequestHandler /
requestHandler name=/analysis/document
class=solr.DocumentAnalysisRequestHandler
startup=lazy /
requestHandler name=/admin/
class=solr.admin.AdminHandlers /
requestHandler name=/admin/ping class=solr.PingRequestHandler
lst name=invariants
str name=qsolrpingquery/str
/lst
lst name=defaults
str name=echoParamsall/str
/lst
/requestHandler
requestHandler name=/debug/dump class=solr.DumpRequestHandler
lst name=defaults
str
Re: unable to load core after cluster restart
--- In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems with
Lucene/Solr.
Shawn, this is interesting. What are the problems, where are the documented?
On 6 November 2013 20:18, kaustubh147 kaustubh.j...@gmail.com wrote:
Hi,
Here is my
solr.xml
solr
solrcloud
str name=host${host:}/str
int name=hostPort28081/int
str name=hostContext/solr/str
str name=zkHostIP1:2181,IP2:2181,IP3:2181/mysolr/str
int name=zkClientTimeout15000/int
bool name=genericCoreNodeNames${genericCoreNodeNames:true}/bool
/solrcloud
shardHandlerFactory name=shardHandlerFactory
class=HttpShardHandlerFactory
int name=socketTimeout${socketTimeout:0}/int
int name=connTimeout${connTimeout:0}/int
/shardHandlerFactory
/solr
--
and
solrconfig.xml
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersion4.5/luceneMatchVersion
lib dir=../../../contrib/extraction/lib regex=.*\.jar /
lib dir=../../../dist/ regex=solr-cell-\d.*\.jar /
lib dir=../../../contrib/clustering/lib/ regex=.*\.jar /
lib dir=../../../dist/ regex=solr-clustering-\d.*\.jar /
lib dir=../../../dist/ regex=solr-dataimporthandler-\d.*\.jar /
lib dir=../../../contrib/dataimporthandler/lib/ regex=.*\.jar /
lib dir=../../../contrib/langid/lib/ regex=.*\.jar /
lib dir=../../../dist/ regex=solr-langid-\d.*\.jar /
lib dir=../../../contrib/velocity/lib regex=.*\.jar /
lib dir=../../../dist/ regex=solr-velocity-\d.*\.jar /
dataDir${solr.data.dir:}/dataDir
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
codecFactory class=solr.SchemaCodecFactory/
schemaFactory class=ClassicIndexSchemaFactory/
indexConfig
writeLockTimeout2/writeLockTimeout
lockType${solr.lock.type:native}/lockType
infoStreamtrue/infoStream
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
updateLog
str name=dir${solr.data.dir:}/str
/updateLog
autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
/autoSoftCommit
/updateHandler
query
maxBooleanClauses50/maxBooleanClauses
filterCache class=solr.FastLRUCache
size=512
initialSize=512
autowarmCount=0/
queryResultCache class=solr.LRUCache
size=512
initialSize=512
autowarmCount=0/
documentCache class=solr.LRUCache
size=512
initialSize=512
autowarmCount=0/
cache name=perSegFilter
class=solr.search.LRUCache
size=10
initialSize=0
autowarmCount=10
regenerator=solr.NoOpRegenerator /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
queryResultWindowSize20/queryResultWindowSize
queryResultMaxDocsCached200/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
/arr
/listener
listener event=firstSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=qstatic firstSearcher warming in solrconfig.xml/str
/lst
/arr
/listener
useColdSearcherfalse/useColdSearcher
maxWarmingSearchers2/maxWarmingSearchers
/query
requestDispatcher handleSelect=false
requestParsers enableRemoteStreaming=true
multipartUploadLimitInKB=2048000
formdataUploadLimitInKB=16384
addHttpRequestToContext=false/
httpCaching never304=true /
/requestDispatcher
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=configdb-data-config.xml/str
/lst
/requestHandler
requestHandler name=/select class=solr.SearchHandler
lst name=defaults
str name=echoParamsexplicit/str
int name=rows10/int
str name=dfkeyword/str
/lst
/requestHandler
requestHandler name=/query class=solr.SearchHandler
lst name=defaults
str name=echoParamsexplicit/str
str name=wtjson/str
str name=indenttrue/str
str name=dfkeyword/str
/lst
/requestHandler
requestHandler name=/get class=solr.RealTimeGetHandler
lst name=defaults
str name=omitHeadertrue/str
str name=wtjson/str
str name=indenttrue/str
/lst
/requestHandler
requestHandler name=/update class=solr.UpdateRequestHandler
/requestHandler
requestHandler name=/update/json class=solr.JsonUpdateRequestHandler
lst name=defaults
str name=stream.contentTypeapplication/json/str
/lst
/requestHandler
requestHandler name=/update/csv class=solr.CSVRequestHandler
lst name=defaults
str name=stream.contentTypeapplication/csv/str
/lst
/requestHandler
requestHandler name=/update/extract
startup=lazy
class=solr.extraction.ExtractingRequestHandler
lst name=defaults
str name=lowernamestrue/str
str name=uprefixignored_/str
str name=captureAttrtrue/str
str name=fmap.alinks/str
str name=fmap.divignored_/str
/lst
/requestHandler
requestHandler name=/analysis/field
startup=lazy
class=solr.FieldAnalysisRequestHandler /
requestHandler name=/analysis/document
Re: unable to load core after cluster restart
On 11/6/2013 2:03 PM, Chris Geeringh wrote: --- In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems with Lucene/Solr. Shawn, this is interesting. What are the problems, where are the documented? https://issues.apache.org/jira/browse/LUCENE-5212 The issue comments say that one of the committers has worked around the problem, but I'm not sure whether that fix has made it out into the wild yet. Even if it has, JVM bugs are nasty and can be tripped by other code besides Lucene/Solr, so it's better to completely avoid Java versions that are known to have them. Thanks, Shawn
Re: unable to load core after cluster restart
On 11/6/2013 1:18 PM, kaustubh147 wrote: I am attaching a small log file with debug option enabled.. log shows following process 1. first start of solr cluster 2. create collection collection1 3. shutdown cluster 4. start cluster again error is in only the 4th step...and it is coming after solr is trying to close data directory for some reason... class name is org.apache.solr.core.CachingDirectoryFactory.. Thanks, Kaustubh solr_logs_for_post.txt http://lucene.472066.n3.nabble.com/file/n4099641/solr_logs_for_post.txt Here's a relevant excerpt from that log: Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/mnt/emc/app_name/data-prod-refresh/SolrCloud/SolrHome3/solr/collection1/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:673) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440) ... 15 more In a nutshell, it's trying to create the write.lock file in your index directory and obtain a lock on it. Either the file already exists from the previous Solr start, or the lock is failing. You may need to change your lock type. Currently it's native but you might need simple or none instead, for NFS to work properly. You might also need to upgrade to NFSv4, which has much better locking capability than earlier versions. Unfortunately it also tends to have mounting problems, leading people to downgrade the version to 3 on their servers. There might be network problems, or bugs in your NFS server or NFS client code, etc. The path suggests that the storage is an EMC device. Check with them for possible bugs with their NFS implementation - you might need newer software/firmware for the device. NIC firmware on your server is another possible problem point. Running with your index on NFS is problematic. I have never personally tried it, but I see messages here and on IRC about problems with it. I believe there are some people who do it successfully, perhaps someone who's faced these problems and beat them could offer insight. Thanks, Shawn
Re: unable to load core after cluster restart
Hi, I tried simple lock type too. It is throwing similar error... Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@/mnt/emc/app_name/data-prod-refresh/SolrCloud/SolrHome1/solr/collection1_shard1_replica2/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:673) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440) I am trying to find if there is anything wrong with NFS/NFS implementation. but similar setup works fine in QA1. I even tried moving my Solrhome from mount drive to server's local drive... but it yield the same error.. Thanks, Kaustubh -- View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099687.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Hi Shawn, One thing I forget to mention here is the same setup (with no bootstrap) is working fine in our QA1 environment. I did not have the bootstrap option from start, I added it thinking it will solve the problem. Nonetheless I followed Shawn's instructions, wherever it differed from my old approach... 1. I moved my zkHost from JVM to solr.xml and added chroot in it 2. removed bootstrap option 3. created collections with URL template suggested (I have tried it earlier too) None of it worked for me... I am seeing same errors.. I am adding some more logs before and after the error occurs - INFO - 2013-11-02 17:40:40.427; org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with IndexWriterCloser INFO - 2013-11-02 17:40:40.428; org.apache.solr.core.SolrCore; [xyz] Closing main searcher on request. INFO - 2013-11-02 17:40:40.431; org.apache.solr.core.CachingDirectoryFactory; Closing NRTCachingDirectoryFactory - 1 directories currently being tracked INFO - 2013-11-02 17:40:40.432; org.apache.solr.core.CachingDirectoryFactory; looking to close /mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data [CachedDirrefCount=0;path=/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data;done=false] INFO - 2013-11-02 17:40:40.432; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data ERROR - 2013-11-02 17:40:40.433; org.apache.solr.core.CoreContainer; Unable to create core: xyz org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.init(SolrCore.java:834) at org.apache.solr.core.SolrCore.init(SolrCore.java:625) at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:555) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:247) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:239) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589) at org.apache.solr.core.SolrCore.init(SolrCore.java:821) ... 13 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:695) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440) ... 15 more ERROR - 2013-11-02 17:40:40.443; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: Unable to create core: xyz at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:934) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:566) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:247) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:239) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.init(SolrCore.java:834)
Re: unable to load core after cluster restart
On 10/31/2013 9:18 PM, kaustubh147 wrote: Glassfish 3.1.2.2 Solr 4.5 Zookeeper 3.4.5 We have set up a SolrCloud with 4 Solr nodes and 3 zookeeper instances. I start the cluster for the first time with bootstrap_conf= true All the nodes starts property.. I am creating cores (with the same name) on all 4 instances. I can add multiple cores on each of the instances... logically I have 5 collections. Now i am creating indexes.. and it automatically creates 4 copies of the index, one for each instance in appropriate SolrHome directory... It will work properly untill I restart the Solr cluster as soon as I restart the cluster, it throws this error (refer below) and none of the collection works properly... It's having problems with the index locking. Further down, you talk about shared directories. I assume you're using a network filesystem, like NFS. Lucene and Solr don't work well with network file systems, and NFS in particular. http://stackoverflow.com/questions/9599529/solr-over-nfs-problems For SolrCloud, using the bootstrap options tends to create a lot of confusion. 1. I have removed dataDir from solrconfig.xml as suggested by Shaun here... http://lucene.472066.n3.nabble.com/Solr-4-3-0-Shard-instances-using-incorrect-data-directory-on-machine-boot-td4063799.html 2. I have provided absolute dataDir path in the core.properties file - https://issues.apache.org/jira/browse/SOLR-4878 3. InstanceDir in each SolrHome have same name for every core/collection-- for example SolrHome1/solr/xyz/conf SolrHome1/solr/xyz/data SolrHome1/solr/xyz/core.properties SolrHome1/solr/pqr/conf SolrHome1/solr/pqr/data SolrHome1/solr/pqr/core.properties SolrHome2/solr/xyz/conf SolrHome2/solr/xyz/data SolrHome2/solr/xyz/core.properties SolrHome2/solr/pqr/conf SolrHome2/solr/pqr/data SolrHome2/solr/pqr/core.properties ... 3. The 4 SolrHome for each of the instances are on a single shared drive... but are in different directories 4. All my collections and cores share the same solrconfig.xml The fact that you're using SolrCloud changes things quite a bit. I notice that you have a conf directory on all your cores. I'm very curious as to why -- because with SolrCloud, the config that's used isn't on the disk, it's in zookeeper ... and when you create a collection from scratch, the cores will NOT contain a conf directory. IMHO, you shouldn't be trying to control things like dataDir with SolrCloud. In fact, you should let SolrCloud control practically all aspects of your cores. Here's some more stuff that's my opinion about how a SolrCloud should be begun and managed: Set the solr home, either with a solr.solr.home java system property or the JNDI property solr/home. In solr.xml, include settings for the hostPort and hostContext, and include a zkHost parameter as well with the following format, where mysolr is a zookeeper chroot and can be anything you want it to be: zoo1.example.com:2181,zoo2.example.com:2181,zoo3.example.com/mysolr Upload one or more configs to zookeeper using the zkcli upconfig command. The first time you start each Solr server, it should have no cores at all, and you should never use any bootstrap options. Once all servers are started, create a collection using the following URL as a template: http://server:port/admin/collections?action=CREATEname=mytestnumShards=XreplicationFactor=Ncollection.configName=mycfg Use appropriate values for name, X and N, and N should be at least 2 unless you're building a dev cluster that doesn't need redundancy. For the collection.configName, use a config name that you uploaded earlier with zkcli. Thanks, Shawn
