Re: Expunge deleting using excessive transient disk space

2013-09-09 Thread Manuel Le Normand
I can only agree for the 50% free space recommendation. Unfortunately I do not have this for the current time, I'm standing on a 10% free disk (out of 300GB for each server). I'm aware it is very low. Does this seem reasonable adapting the current merge policy (or writing a new one) that would

Re: Expunge deleting using excessive transient disk space

2013-09-09 Thread Walter Underwood
10% free space is guaranteed to cause problems. That is a faulty installation. Explain to ops that Solr needs double the minimum index size. This is required for normal operation. That isn't extra, it is required for merges. Solr makes copies instead of doing record locking. The merge design is

Re: Expunge deleting using excessive transient disk space

2013-09-09 Thread Chris Hostetter
: Looking on the infostream I can see that the first merges do succeed but : older segments are kept in reference thus cannot be deleted until all the : merging are done. I suspect what you are seeing is that filehandles for the older segemnts are kept open (and thus, the bytes on disk for

Expunge deleting using excessive transient disk space

2013-09-08 Thread Manuel Le Normand
Hi, In order to delete part of my index I run a delete by query that intends to erase 15% of the docs. I added this params to the solrconfig.xml mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce2/int int name=maxMergeAtOnceExplicit2/int double

Re: Expunge deleting using excessive transient disk space

2013-09-08 Thread Erick Erickson
Right, but you should have at least as much free space as your total index size, and I don't see the total index size (but I'm just glancing). I'm not entirely sure you can precisely calculate the maximum free space you have relative to the amount needed for merging, some of the people who wrote