Re: What can cause segment corruption?

2011-01-12 Thread Michael McCandless 
Corruption should only happen if 1) we have a bug in Lucene (but we
work hard to fix such bugs, though, LUCENE-2593, fixed in 2.9.4, is a
recent case) or 2) there are hardware problems on the machine.

Mike

On Tue, Jan 11, 2011 at 10:02 AM, Stéphane Delprat
stephane.delp...@blogspirit.com wrote:
 Thanks for your answer,

 It's not a disk space problem here :

 # df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/sda4             280G   22G  244G   9% /


 We will try to install solr on a different server (We just need a little
 time for that)


 Stéphane


 Le 11/01/2011 15:42, Jason Rutherglen a écrit :

 Stéphane,

 I've only seen production index corruption when during merge the
 process ran out of disk space, or there is an underlying hardware
 related issue.

 On Tue, Jan 11, 2011 at 5:06 AM, Stéphane Delprat
 stephane.delp...@blogspirit.com  wrote:

 Hi,


 I'm using Solr 1.4.1 (Lucene 2.9.3)

 And some segments get corrupted:

  4 of 11: name=_p40 docCount=470035
    compound=false
    hasProx=true
    numFiles=9
    size (MB)=1,946.747
    diagnostics = {optimize=true, mergeFactor=6,
 os.version=2.6.26-2-amd64,
 os=Linux, mergeDocStores=true, lucene.version=2.9.3 951790 - 2010-06-06
 01:30:55, source=merge, os.arch=amd64, java.version=1.6.0_20,
 java.vendor=Sun Microsystems Inc.}
    has deletions [delFileName=_p40_bj.del]
    test: open reader.OK [9299 deleted docs]
    test: fields..OK [51 fields]
    test: field norms.OK [51 fields]
    test: terms, freq, prox...ERROR [term source:margolisphil docFreq=1 !=
 num docs seen 0 + num docs deleted 0]
 java.lang.RuntimeException: term source:margolisphil docFreq=1 != num
 docs
 seen 0 + num docs deleted 0
        at
 org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:675)
        at
 org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
    test: stored fields...OK [15454281 total field count; avg 33.543
 fields per doc]
    test: term vectorsOK [0 total vector count; avg 0 term/freq
 vector fields per doc]
 FAILED
    WARNING: fixIndex() would remove reference to this segment; full
 exception:
 java.lang.RuntimeException: Term Index test failed
        at
 org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:543)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)


 What might cause this corruption?


 I detailed my configuration here:


 http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201101.mbox/%3c4d2ae506.7070...@blogspirit.com%3e

 Thanks,

What can cause segment corruption?

2011-01-11 Thread Stéphane Delprat 

Hi,


I'm using Solr 1.4.1 (Lucene 2.9.3)

And some segments get corrupted:

  4 of 11: name=_p40 docCount=470035
compound=false
hasProx=true
numFiles=9
size (MB)=1,946.747
diagnostics = {optimize=true, mergeFactor=6, 
os.version=2.6.26-2-amd64, os=Linux, mergeDocStores=true, 
lucene.version=2.9.3 951790 - 2010-06-06 01:30:55, source=merge, 
os.arch=amd64, java.version=1.6.0_20, java.vendor=Sun Microsystems Inc.}

has deletions [delFileName=_p40_bj.del]
test: open reader.OK [9299 deleted docs]
test: fields..OK [51 fields]
test: field norms.OK [51 fields]
test: terms, freq, prox...ERROR [term source:margolisphil docFreq=1 
!= num docs seen 0 + num docs deleted 0]
java.lang.RuntimeException: term source:margolisphil docFreq=1 != num 
docs seen 0 + num docs deleted 0
at 
org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:675)
at 
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)

at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
test: stored fields...OK [15454281 total field count; avg 
33.543 fields per doc]
test: term vectorsOK [0 total vector count; avg 0 term/freq 
vector fields per doc]

FAILED
WARNING: fixIndex() would remove reference to this segment; full 
exception:

java.lang.RuntimeException: Term Index test failed
at 
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:543)

at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)


What might cause this corruption?


I detailed my configuration here:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201101.mbox/%3c4d2ae506.7070...@blogspirit.com%3e

Thanks,

Re: What can cause segment corruption?

2011-01-11 Thread Jason Rutherglen 
Stéphane,

I've only seen production index corruption when during merge the
process ran out of disk space, or there is an underlying hardware
related issue.

On Tue, Jan 11, 2011 at 5:06 AM, Stéphane Delprat
stephane.delp...@blogspirit.com wrote:
 Hi,


 I'm using Solr 1.4.1 (Lucene 2.9.3)

 And some segments get corrupted:

  4 of 11: name=_p40 docCount=470035
    compound=false
    hasProx=true
    numFiles=9
    size (MB)=1,946.747
    diagnostics = {optimize=true, mergeFactor=6, os.version=2.6.26-2-amd64,
 os=Linux, mergeDocStores=true, lucene.version=2.9.3 951790 - 2010-06-06
 01:30:55, source=merge, os.arch=amd64, java.version=1.6.0_20,
 java.vendor=Sun Microsystems Inc.}
    has deletions [delFileName=_p40_bj.del]
    test: open reader.OK [9299 deleted docs]
    test: fields..OK [51 fields]
    test: field norms.OK [51 fields]
    test: terms, freq, prox...ERROR [term source:margolisphil docFreq=1 !=
 num docs seen 0 + num docs deleted 0]
 java.lang.RuntimeException: term source:margolisphil docFreq=1 != num docs
 seen 0 + num docs deleted 0
        at
 org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:675)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
    test: stored fields...OK [15454281 total field count; avg 33.543
 fields per doc]
    test: term vectorsOK [0 total vector count; avg 0 term/freq
 vector fields per doc]
 FAILED
    WARNING: fixIndex() would remove reference to this segment; full
 exception:
 java.lang.RuntimeException: Term Index test failed
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:543)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)


 What might cause this corruption?


 I detailed my configuration here:

 http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201101.mbox/%3c4d2ae506.7070...@blogspirit.com%3e

 Thanks,

Re: What can cause segment corruption?

2011-01-11 Thread Stéphane Delprat 

Thanks for your answer,

It's not a disk space problem here :

# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sda4 280G   22G  244G   9% /


We will try to install solr on a different server (We just need a little 
time for that)



Stéphane


Le 11/01/2011 15:42, Jason Rutherglen a écrit :

Stéphane,

I've only seen production index corruption when during merge the
process ran out of disk space, or there is an underlying hardware
related issue.

On Tue, Jan 11, 2011 at 5:06 AM, Stéphane Delprat
stephane.delp...@blogspirit.com  wrote:

Hi,


I'm using Solr 1.4.1 (Lucene 2.9.3)

And some segments get corrupted:

  4 of 11: name=_p40 docCount=470035
compound=false
hasProx=true
numFiles=9
size (MB)=1,946.747
diagnostics = {optimize=true, mergeFactor=6, os.version=2.6.26-2-amd64,
os=Linux, mergeDocStores=true, lucene.version=2.9.3 951790 - 2010-06-06
01:30:55, source=merge, os.arch=amd64, java.version=1.6.0_20,
java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_p40_bj.del]
test: open reader.OK [9299 deleted docs]
test: fields..OK [51 fields]
test: field norms.OK [51 fields]
test: terms, freq, prox...ERROR [term source:margolisphil docFreq=1 !=
num docs seen 0 + num docs deleted 0]
java.lang.RuntimeException: term source:margolisphil docFreq=1 != num docs
seen 0 + num docs deleted 0
at
org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:675)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
test: stored fields...OK [15454281 total field count; avg 33.543
fields per doc]
test: term vectorsOK [0 total vector count; avg 0 term/freq
vector fields per doc]
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.lang.RuntimeException: Term Index test failed
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:543)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)


What might cause this corruption?


I detailed my configuration here:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201101.mbox/%3c4d2ae506.7070...@blogspirit.com%3e

Thanks,