[ https://issues.apache.org/jira/browse/LUCENE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292536#comment-14292536 ]
Tom Burton-West commented on LUCENE-6192: ----------------------------------------- Patch works! Thanks Mike! Deployed Solr war with the patch and ran optimize on 12 shards. All CheckIndexes passed. Below are some of the stats on one of the shards. Tom About 1 million docs and 700GB index with about 4 billion unique terms, 270 billion tokens docCount=1086381 size (MB)=693,308.47 test: terms, freq, prox...OK [4113882974 terms; 61631126560 terms/docs pairs; 270670957886 tokens] field "ocr": index FST: 27157406 nodes 77300582 arcs 1262090664 bytes terms: 4087713620 terms 50227574334 bytes (12.3 bytes/term) blocks: 132202225 blocks 96419097 terms-only blocks 40757 sub-block-only blocks 35742371 mixed blocks 27202047 floor blocks 44718055 non-floor blocks 87484170 floor sub-blocks 23560113026 term suffix bytes (178.2 suffix-bytes/block) 8227225977 term stats bytes (62.2 stats-bytes/block) 19664735257 other bytes (148.7 other-bytes/block) by prefix length: > Long overflow in LuceneXXSkipWriter can corrupt skip data > --------------------------------------------------------- > > Key: LUCENE-6192 > URL: https://issues.apache.org/jira/browse/LUCENE-6192 > Project: Lucene - Core > Issue Type: Bug > Reporter: Michael McCandless > Assignee: Michael McCandless > Fix For: 5.0, Trunk, 4.x > > Attachments: LUCENE-6192.patch > > > I've been iterating with Tom on this corruption that CheckIndex detects in > his rather large index (720 GB in a single segment): > {noformat} > java -Xmx16G -Xms16G -cp $JAR -ea:org.apache.lucene... > org.apache.lucene.index.CheckIndex /XXXX/shards/4/core-1/data/test_index > -verbose 2>&1 |tee -a shard4_reoptimizedNewJava > Opening index @ /htsolr/lss-reindex/shards/4/core-1/data/test_index > Segments file=segments_e numSegments=1 version=4.10.2 format= > userData={commitTimeMSec=1421479358825} > 1 of 1: name=_8m8 docCount=1130856 > version=4.10.2 > codec=Lucene410 > compound=false > numFiles=10 > size (MB)=719,967.32 > diagnostics = {timestamp=1421437320935, os=Linux, > os.version=2.6.18-400.1.1.el5, mergeFactor=2, source=merge, > lucene.version=4.10.2, os.arch=amd64, mergeMaxNumSegments=1, > java.version=1.7.0_71, java.vendor=Oracle Corporation} > no deletions > test: open reader.........OK > test: check integrity.....OK > test: check live docs.....OK > test: fields..............OK [80 fields] > test: field norms.........OK [23 fields] > test: terms, freq, prox...ERROR: java.lang.AssertionError: -96 > java.lang.AssertionError: -96 > at > org.apache.lucene.codecs.lucene41.ForUtil.skipBlock(ForUtil.java:228) > at > org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.skipPositions(Lucene41PostingsReader.java:925) > at > org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextPosition(Lucene41PostingsReader.java:955) > at > org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:1100) > at > org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1357) > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:655) > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2096) > test: stored fields.......OK [67472796 total field count; avg 59.665 > fields per doc] > test: term vectors........OK [0 total vector count; avg 0 term/freq > vector fields per doc] > test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 > SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] > FAILED > WARNING: fixIndex() would remove reference to this segment; full > exception: > java.lang.RuntimeException: Term Index test failed > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:670) > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2096) > WARNING: 1 broken segments (containing 1130856 documents) detected > WARNING: would write new segments file, and 1130856 documents would be lost, > if -fix were specified > {noformat} > And Rob spotted long -> int casts in our skip list writers that look like > they could cause such corruption if a single high-freq term with many > positions required > 2.1 GB to write its positions into .pos. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org