UPDATE:
Crazy staff with SLES10 SP2 default installation/partitioning, LVM (Logical
Volume Manager) shows 400Gb available, but... I lost 90% of index without
even noticing that!
Aug 16, 2009 8:04:32 PM org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java)
- then somehow no any exceptions without few hours, no any corrupted index
after several commits, then again "not enough space", etc.; finally
corrupted index (still, SATA)
Thanks
-----Original Message-----
From: Funtick [mailto:f...@efendi.ca]
Sent: August-18-09 12:25 AM
To: solr-user@lucene.apache.org
Subject: Re: SOLR <uniqueKey> - extremely strange behavior! Documents
disappeared...
sorry for typo in prev msg,
Increase = 2,297,231 - 1,786,552 = 500,000 (average)
RATE (non-unique-id:unique-id) = 7,000,000 : 500,000 = 14:1
but 125:1 (initial 30 hours) was very strange...
Funtick wrote:
>
> UPDATE:
>
> After few more minutes (after previous commit):
> docsPending: about 7,000,000
>
> After commit:
> numDocs: 2,297,231
>
> Increase = 2,297,231 - 1,281,851 = 1,000,000 (average)
>
> So that I have 7 docs with same ID in average.
>
> Having 100,000,000 and then dropping below 1,000,000 is strange; it is a
> bug somewhere... need to investigate ramBufferSize and MergePolicy,
> including SOLR uniqueId implementation...
>
>
>
> Funtick wrote:
>>
>> After running an application which heavily uses MD5 HEX-representation as
>> <uniqueKey> for SOLR v.1.4-dev-trunk:
>>
>> 1. After 30 hours:
>> 101,000,000 documents added
>>
>> 2. Commit:
>> numDocs = 783,714
>> maxDoc = 3,975,393
>>
>> 3. Upload new docs to SOLR during 1 hour(!!!!!!!), then commit, then
>> optimize:
>> numDocs=1,281,851
>> maxDocs=1,281,851
>>
>> It looks _extremely_ strange that within an hour I have such a huge
>> increase with same 'average' document set...
>>
>> I am suspecting something goes wrong with Lucene buffer flush / index
>> merge OR SOLR - Unique ID handling...
>>
>> According to my own estimates, I should have about 10,000,000 new
>> documents now... I had 0.5 millions within an hour, and 0.8 mlns within a
>> day; same 'random' documents.
>>
>> This morning index size was about 4Gb, then suddenly dropped below 0.5
>> Gb. Why? I haven't issued any "commit"...
>>
>> I am using ramBufferMB=8192
>>
>>
>>
>>
>>
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/SOLR-%3CuniqueKey%3E---extremely-strange-behavior%21-D
ocuments-disappeared...-tp25017728p25018263.html
Sent from the Solr - User mailing list archive at Nabble.com.
