Thank you all for the reply and shedding more light on this topic.
A follow up question: during optimization, If I run out of disk space, what
happens other than the optimizer failing? Am I now left with even a larger
index than I started with or am I back to the original none optimized index
size?!!!
-- MJ
-----Original Message-----
From: Walter Underwood <wun...@wunderwood.org>
To: solr-user <solr-user@lucene.apache.org>
Sent: Thu, Jun 26, 2014 10:50 am
Subject: Re: How much free disk space will I need to optimize my index
The 3x worst case is:
1. All documents are in one segment.
2. Without merging, all documents are deleted, then re-added and committed.
3. A merge is done.
At the end of step 2, there are two equal-sized segments, 2X the space needed.
During step 3, a third segment of that size is created.
This can only happen if you disable merging. 2X is a conservative margin that
should work fine for regular merges.
Forced full merges ("optimize") can use more overhead because they move every
document in the index. Yet another reason to avoid forced merges.
wunder
On Jun 26, 2014, at 12:50 AM, Thomas Egense <thomas.ege...@gmail.com> wrote:
> That is correct, but twice the disk space is theoretically not enough.
> Worst case is actually three times the storage, I guess this worst case can
> happen if you also submit new documents to the index while optimizing.
> I have experienced 2.5 times the disk space during an optimize for a large
> index, it was a 1TB index that temporarily used 2.5TB disc space during the
> optimize (near the end of the optimization).
>
> From,
> Thomas Egense
>
>
> On Wed, Jun 25, 2014 at 8:21 PM, Markus Jelsma <markus.jel...@openindex.io>
> wrote:
>
>>
>>
>>
>>
>> -----Original message-----
>>> From:johnmu...@aol.com <johnmu...@aol.com>
>>> Sent: Wednesday 25th June 2014 20:13
>>> To: solr-user@lucene.apache.org
>>> Subject: How much free disk space will I need to optimize my index
>>>
>>> Hi,
>>>
>>>
>>> I need to de-fragment my index. My question is, how much free disk
>> space I need before I can do so? My understanding is, I need 1X free disk
>> space of my current index un-optimized index size before I can optimize it.
>> Is this true?
>>
>> Yes, 20 GB of FREE space to force merge an existing 20 GB index.
>>
>>>
>>>
>>> That is, let say my index is 20 GB (un-optimized) then I must have 20 GB
>> of free disk space to make sure the optimization is successful. The reason
>> for this is because during optimization the index is re-written (is this
>> the case?) and if it is already optimized, the re-write will create a new
>> 20 GB index before it deletes the old one (is this true?), thus why there
>> must be at least 20 GB free disk space.
>>>
>>>
>>> Can someone help me with this or point me to a wiki on this topic?
>>>
>>>
>>> Thanks!!!
>>>
>>>
>>> - MJ
>>>
>>
--
Walter Underwood
wun...@wunderwood.org
