Marcin is correct. The index size on disk will perhaps double. (triple in compound case). The reason is so you don't lose your index if the process is interrupted.
Consider the case where you're optimizing to one segment. 1> All the current segments are copied into the new segment 2> The new segment is flushed 3> "control files" that tell Lucene what files constitute the valid segment(s) are written. 4> the old segments are removed. So at any point up to <3> if the system is killed, crashes, whatever, then the old version of the index is intact and you can keep on working, even optimizing again. If, on the other hand, after each segment was written to the new segment the old segment was deleted, interrupting the process (which may be very long) would leave your index in an inconsistent state. FWIW, Erick On Tue, May 20, 2014 at 4:14 AM, Marcin Rzewucki <mrzewu...@gmail.com> wrote: > As I wrote before index is being rewritten so it grows during optimization > and later is reduced. I guess there was OOM in your case. > > > > On 20 May 2014 12:11, YouPeng Yang <yypvsxf19870...@gmail.com> wrote: > >> Hi >> My DIH work indeed hangs, I have only four shards,each has a master and a >> replica.Maybe jvm memory size is very low.it was 3G while the size of >> every >> my core is almost 16GB. >> >> I also have found that the size of the master increased during the >> optimization(you can check on the overview page of the core.).the >> phenomenon is very werid. Is it because that the collection overall >> optimization will comput and copy all the docs of the whole collection. >> >> >> Version Gen Size Master (Searching) >> 1400501330248 >> 98396 >> 29.83 GB >> Master (Replicable) >> 1400501330888 >> 98397 >> - >> >> >> After I have check source code,unfortunatly,it seems the optimize action >> distrib overall the collection.you can reference the >> SolrCmdDistributor.distribCommit. >> >> >> 2014-05-20 17:27 GMT+08:00 Marcin Rzewucki <mrzewu...@gmail.com>: >> >> > Well, it should not hang if all is configured fine :) How many shards and >> > memory you have ? Note that optimize rewrites index so you might need >> > additional disk space for this process. Optimizing works fine however I'd >> > like to be able to do it on a single shard as well. >> > >> > >> > On 20 May 2014 11:19, YouPeng Yang <yypvsxf19870...@gmail.com> wrote: >> > >> > > Hi Marcin >> > > >> > > Thanks to your mail,now I know why my cloud hangs when I just click >> the >> > > optimize button on the overview page of the shard. >> > > >> > > >> > > 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <iori...@yahoo.com>: >> > > >> > > > Hi Marcin, >> > > > >> > > > just a guess, pass distrib=false ? >> > > > >> > > > >> > > > >> > > > Ahmet >> > > > >> > > > >> > > > On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki < >> > mrzewu...@gmail.com> >> > > > wrote: >> > > > Hi, >> > > > >> > > > Do you know how to optimize index on a single shard only ? I was >> trying >> > > to >> > > > use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not >> > > work >> > > > - it optimizes all shards instead of just one. >> > > > >> > > > Kind regards. >> > > > >> > > > >> > > >> > >>
