[
https://issues.apache.org/jira/browse/LUCENE-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594225#action_12594225
]
Michael McCandless commented on LUCENE-1278:
--------------------------------------------
It looks like the .tii file is also storing the int[] docIDs (as
inlined byte blob)? I think that shouldn't be necessary?
This change adds a posting list like the frq file, except that it
stores only docIDs (no freq information), is stored inline in the term
dict, and includes a reader that materializes the full doc list as an
int[] instead of offering an iterator like (nextDoc()) interface
alone.
I think these changes would fit cleanly into what's been proposed for
flexible indexing. EG, case 1a talks about storing only docID in a
posting list, here:
http://wiki.apache.org/jakarta-lucene/FlexibleIndexing
And recent discussions on the dev list around how to be flexible as to
which index file(s) (one or many) things are stored in, eg:
http://www.mail-archive.com/java-dev@lucene.apache.org/msg15681.html
should allow you to store this data inlined into the terms dict, or as
a separate file.
Some other initial comments/questions:
* I think this would bloat the index because the docIDs are being
double stored (in the terms dict, and, in the frq file). Would
you propose changing the frq file to not store the docID when the
term dict is doing so?
* Why store the byte blob in the term dict, and not a separate (new)
index file? We lose locality for cases where one wants to iterate
through terms and not loads these docs (eg RangeQuery).
* Could you, instead, make a reader that reads in the full byte blob
from the frq file for a term, and then processes that into the
int[]? This would require no change to indexing & the index
format, and wouldn't waste space double-storing the docIDs.
* I'm worried how well this scales up. For very common terms
allocating then decoding & holding entirely in RAM the full list
of docIDs can become extremely costly. Also, I don't have a clear
sense of how apps would use the returned int[]. For example,
would the int[] for many terms need to remain resident at the same
time? (Eg when running a RangeQuery). If so, that compounds the
scale challenge.
> Add optional storing of document numbers in term dictionary
> -----------------------------------------------------------
>
> Key: LUCENE-1278
> URL: https://issues.apache.org/jira/browse/LUCENE-1278
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Affects Versions: 2.3.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Attachments: lucene.1278.5.4.2008.patch, lucene.1278.5.5.2008.patch
>
>
> Add optional storing of document numbers in term dictionary. String index
> field cache and range filter creation will be faster.
> Example read code:
> {noformat}
> TermEnum termEnum = indexReader.terms(TermEnum.LOAD_DOCS);
> do {
> Term term = termEnum.term();
> if (term == null || term.field() != field) break;
> int[] docs = termEnum.docs();
> } while (termEnum.next());
> {noformat}
> Example write code:
> {noformat}
> Document document = new Document();
> document.add(new Field("tag", "dog", Field.Store.YES,
> Field.Index.UN_TOKENIZED, Field.Term.STORE_DOCS));
> indexWriter.addDocument(document);
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
