Hi Brent,
Please see inline comments.
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 26.08.2016 04:51, Brent P wrote:
I'm trying to set up a Solr Cloud cluster to support a system with the
following characteristics:
It will be writing documents at a rate of approximately 500 docs/second,
and running search queries at about the same rate.
The documents are fairly small, with about 10 fields, most of which range
in size from a simple int to a string that holds a UUID. There's a date
field, and then three text fields that typically hold in the range of 350
to 500 chars.
There should be no problems with ingestion on 24 machines. Assuming 1
replication, that is roughly 40 doc/sec/server. Make sure you bulk docs
when ingesting.
Documents should be available for searching within 30 seconds of being
added.
Make sure you don't do explicit commits and use only auto commit.
We need an average search latency of 50 ms or faster.
What is the number of documents you expect? What type of queries do you
have? Make sure you use filters wherever possible.
We've been using DataStax Enterprise with decent results, but trying to
determine if we can get more out of the latest version of Solr Cloud, as we
originally chose DSE ~4 years ago *I believe* because its Cassandra-backed
Solr provided redundancy/high availability features that weren't currently
available with straight Solr (not even sure if Solr Cloud was available
then).
We have 24 fairly beefy servers (96 CPU cores, 256 GB RAM, SSDs) for the
task, and I'm trying to figure out the best way to distribute the documents
into collections, cores, and shards.
If I can categorize a document into one of 8 "types", should I create 8
collections? Is that going to provide better performance than putting them
all into one collection and then using a filter query with the type field
when doing a search?
If you don't need to share term frequencies between types and if you
always search one type, I would split collections. Set up number of
shards for each collection according to number of docs in it.
Alternatively, you could use routing by type or in case you need to
split to more than one shard, you can use composite hash routing.
(https://sematext.com/blog/2015/09/29/solrcloud-large-tenants-and-routing/).
What are the options/things to consider when deciding on the number of
shards for each collection?
Number of documents and query latency are main factors in determining
number of shards. Smaller the shard, faster the query, but more shards
are queried, more data to merge so at one point benefits of distributed
query are eaten be overhead of merging.
As far as I know, I don't choose the number of
Solr cores, that is just determined base on the replication factor (and
shard count?).
Some of the settings I'm using in my solrconfig that seem important:
<lockType>${solr.lock.type:native}</lockType>
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:30000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
This is not how soon your changes will be visible (openSearcher=false).
This is how frequent your modifications will be flushed.
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:1000}</maxTime>
This is how soon you need your changes will be visible and should be set
to 30s (or the highest possible value since caches are invalidated when
searcher is opened.
</autoSoftCommit>
<useColdSearcher>true</useColdSearcher>
<maxWarmingSearchers>8</maxWarmingSearchers>
You can keep these setting but they are hiding configuration errors. It
is better to have those errors and fix warming configs than using cold
and allowing large number of warming searchers.
I've got the updateLog/transaction log enabled, as I think I read it's
required for Solr Cloud.
Are there any settings I should look at that affect performance
significantly, especially outside of the solrconfig.xml for each collection
(like jetty configs, logging properties, etc)?
How much impact do the <lib/> directives in the solrconfig have on
performance? Do they only take effect if I have something configured that
requires them, and therefore if I'm missing one that I need, I'd get an
error if it's not defined?
Any help will be greatly appreciated. Thanks!
-Brent
