I upgraded Solr from 5.2.1 to 5.5.4 recently. Occasionally when creating a new collection via the Collections API I get an error: Could not fully create collection: <name>. This has never happened previous to this upgrade.
It's happened twice in my development environment and once in my user test environment. These are small 2 node clusters with less than a dozen total collections. What I see in the admin console is a collection with one shard and no replicas. A partial log output is below (cut n paste from Splunk). I am not manually typing a create command, I have a custom UI that handles all necessary administration via SolrJ. Is there a known issue here or am I inadvertently causing this problem? 2017-12-21 15:35:46.836 INFO org.apache.solr.cloud.OverseerTaskProcessor$Runner Overseer Collection Message Handler: Message id:/overseer/collection-queue-work/qn-0001184876 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Could not fully create collection: ANewCollection,exception={msg=Could not fully create collection: ANewCollection,rspCode=500}} host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:46.684 ERROR org.apache.solr.common.SolrException Collection: ANewCollection operation: create failed:org.apache.solr.common.SolrException: Could not fully create collection: ANewCollection at org.apache.solr.cloud.OverseerCollectionMessageHandler.createCollection(OverseerCollectionMessageHandler.java:1993) at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:221) at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:437) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.227 INFO org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=11 host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.227 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection1] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.226 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection1] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.226 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection2] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.226 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection2] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.225 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection3] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.225 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection3] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.225 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection4] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.224 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection4] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.224 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection5] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.223 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection5] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.223 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection6] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.223 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection6] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.223 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection7] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.222 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection7] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.222 INFO org.apache.solr.common.cloud.ZkStateReader path=[/collections/Collection8] [configName]=[default] specified config exists in ZooKeeper host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.221 INFO org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/Collection8] host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:29.216 INFO org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.576 INFO org.apache.solr.common.cloud.ZkStateReader$CollectionsChildWatcher A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating... host=solr1.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk5.server.com Options 2017-12-21 15:35:16.572 INFO org.apache.solr.common.cloud.SolrZkClient makePath: /collections/ANewCollection host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.571 INFO org.apache.solr.cloud.OverseerCollectionMessageHandler creating collections conf node /collections/ANewCollection host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.562 INFO org.apache.solr.cloud.OverseerCollectionMessageHandler Create collection ANewCollection host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.553 INFO org.apache.solr.cloud.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , { "name":"ANewCollection", "fromApi":"true", "replicationFactor":"2", "collection.configName":"default", "numShards":"1", "maxShardsPerNode":"1", "createNodeSet":"solr2.server.com:8983_solr,solr1.server.com:8983_solr", "createNodeSet.shuffle":"false", "stateFormat":"2", Show all 11 lines host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.553 INFO org.apache.solr.cloud.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0001184876 message:{ "name":"ANewCollection", "fromApi":"true", "replicationFactor":"2", "collection.configName":"default", "numShards":"1", "maxShardsPerNode":"1", "createNodeSet":"solr2.server.com:8983_solr,solr1.server.com:8983_solr", "createNodeSet.shuffle":"false", "stateFormat":"2", Show all 11 lines host=solr2.server.com Options| sourcetype=solr Options| source=/var/log/solr/solr.log Options| splunk_server=splunk2.server.com Options 2017-12-21 15:35:16.244 INFO org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=default&maxShardsPerNode=1&name=ANewCollection&action=CREATE&numShards=1&createNodeSet=solr2.server.com:8983_solr,solr1.server.com:8983_solr&createNodeSet.shuffle=false&wt=json and sendToOCPQueue=true -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html