On 9/12/13 4:26 PM, Shawn Heisey wrote:
On 9/12/2013 2:14 AM, Per Steffensen wrote:
Starting from an empty collection. Things are fine wrt
storing/indexing speed for the first two-three hours (100M docs per
hour), then speed goes down dramatically, to an, for us, unacceptable
level (max 10M per hour). At the same time as speed goes down, we see
that I/O wait increases dramatically. I am not 100% sure, but quick
investigation has shown that this is due to almost constant merging.
While constant merging is contributing to the slowdown, I would guess
that your index is simply too big for the amount of RAM that you have.
Let's ignore for a minute that you're distributed and just concentrate
on one machine.
After three hours of indexing, you have nearly 300 million documents.
If you have a replicationFactor of 1, that's still 50 million documents
per machine. If your replicationFactor is 2, you've got 100 million
documents per machine. Let's focus on the smaller number for a minute.
replicationFactor is 1, so that is about 50 million docs per machine at
this point
50 million documents in an index, even if they are small documents, is
probably going to result in an index size of at least 20GB, and quite
possibly larger. In order to make Solr function with that many
documents, I would guess that you have a heap that's at least 4GB in size.
Currently I have 2,5GB heap, on the 8GB machine - to leave something for
the OS cache
With only 8GB on the machine, this doesn't leave much RAM for the OS
disk cache. If we assume that you have 4GB left for caching, then I
would expect to see problems about the time your per-machine indexes hit
15GB in size. If you are making it beyond that with a total of 300
million documents, then I am impressed.
Two things are going to happen when you have enough documents: 1) You
are going to fill up your Java heap and Java will need to do frequent
collections to free up enough RAM for normal operation. When this
problem gets bad enough, the frequent collections will be *full* GCs,
which are REALLY slow.
What is it that will fill my heap? I am trying to avoid the FieldCache.
For now, I am actually not doing any searches - focus on indexing for
now - and certainly not group/facet/sort searches that will use the
FieldCache.
2) The index will be so big that the OS disk
cache cannot effectively cache it. I suspect that the latter is more of
the problem, but both might be happening at nearly the same time.
When dealing with an index of this size, you want as much RAM as you can
possibly afford. I don't think I would try what you are doing without
at least 64GB per machine, and I would probably use at least an 8GB heap
on each one, quite possibly larger. With a heap that large, extreme GC
tuning becomes a necessity.
More RAM will probably help, but only for a while. I want billions of
documents in my collections - and also on each machine. Currently we are
aiming 15 billion documents per month (500 million per day) and keep at
least two years of data in the system. Currently we use one collection
for each month, so when the system has been running for two years it
will be 24 collections with 15 billion documents each. Indexing will
only go on in the collection corresponding to the "current" month, but
searching will (potentially) be across all 24 collections. The documents
are very small. I know that 6 machines will not do in the long run -
currently this is only testing - but number of machines should not be
higher than about 20-40. In general it is a problem if Solr/Lucene will
not perform fairly well if data does not fit RAM - then it cannot really
be used for "big data". I would have to buy hundreds or even thousands
of machines with 64GB+ RAM. That is not realistic.
To cut down on the amount of merging, I go with a fairly large
mergeFactor, but mergeFactor is basically deprecated for
TieredMergePolicy, there's a new way to configure it now. Here's the
indexConfig settings that I use on my dev server:
<indexConfig>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">35</int>
<int name="segmentsPerTier">35</int>
<int name="maxMergeAtOnceExplicit">105</int>
</mergePolicy>
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">1</int>
<int name="maxMergeCount">6</int>
</mergeScheduler>
<ramBufferSizeMB>48</ramBufferSizeMB>
<infoStream file="INFOSTREAM-${solr.core.name}.txt">false</infoStream>
</indexConfig>
Thanks,
Shawn
Thanks!
