[ https://issues.apache.org/jira/browse/LUCENE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935977#action_12935977 ]
Michael McCandless commented on LUCENE-2186: -------------------------------------------- bq. BTW. it is ok to have the same name as a existing field. It is, usually... but we should add a test to assert this is still the case for other field + ValuesField? {quote} bq. I'm thinking it's really important now to carry over the same FieldInfos from the last segment when opening the writer (LUCENE-1737)... because hitting that IllegalStateExc during merge is a trap. I think that should not block us from moving forward and landing on trunk ey? {quote} It makes me mighty nervous though... I'll try to get that issue done soon. {quote} Well it is a nice way of extending field but I am not sure if we should keep it since it is heavy weight. {quote} The ValuesAttr for ValuesField is actually really heavyweight. Not only must it fire up an AttrSource, but then ValuesAttrImpl itself has a field for each type. Worse, for the type you do actually use, it's then another object eg FloatsRef, which in turn holds array/offset/len, a new length 1 array, etc. Maybe we shouldn't use attrs here? And instead somehow let ValuesField store a single value as it's own private member? FloatsRef, LongsRef are missing the ASL header. Maybe it's time to run RAT :) > First cut at column-stride fields (index values storage) > -------------------------------------------------------- > > Key: LUCENE-2186 > URL: https://issues.apache.org/jira/browse/LUCENE-2186 > Project: Lucene - Java > Issue Type: New Feature > Components: Index > Reporter: Michael McCandless > Assignee: Simon Willnauer > Fix For: CSF branch, 4.0 > > Attachments: LUCENE-2186.patch, LUCENE-2186.patch, LUCENE-2186.patch, > LUCENE-2186.patch, LUCENE-2186.patch, mem.py > > > I created an initial basic impl for storing "index values" (ie > column-stride value storage). This is still a work in progress... but > the approach looks compelling. I'm posting my current status/patch > here to get feedback/iterate, etc. > The code is standalone now, and lives under new package > oal.index.values (plus some util changes, refactorings) -- I have yet > to integrate into Lucene so eg you can mark that a given Field's value > should be stored into the index values, sorting will use these values > instead of field cache, etc. > It handles 3 types of values: > * Six variants of byte[] per doc, all combinations of fixed vs > variable length, and stored either "straight" (good for eg a > "title" field), "deref" (good when many docs share the same value, > but you won't do any sorting) or "sorted". > * Integers (variable bit precision used as necessary, ie this can > store byte/short/int/long, and all precisions in between) > * Floats (4 or 8 byte precision) > String fields are stored as the UTF8 byte[]. This patch adds a > BytesRef, which does the same thing as flex's TermRef (we should merge > them). > This patch also adds basic initial impl of PackedInts (LUCENE-1990); > we can swap that out if/when we get a better impl. > This storage is dense (like field cache), so it's appropriate when the > field occurs in all/most docs. It's just like field cache, except the > reading API is a get() method invocation, per document. > Next step is to do basic integration with Lucene, and then compare > sort performance of this vs field cache. > For the "sort by String value" case, I think RAM usage & GC load of > this index values API should be much better than field caache, since > it does not create object per document (instead shares big long[] and > byte[] across all docs), and because the values are stored in RAM as > their UTF8 bytes. > There are abstract Writer/Reader classes. The current reader impls > are entirely RAM resident (like field cache), but the API is (I think) > agnostic, ie, one could make an MMAP impl instead. > I think this is the first baby step towards LUCENE-1231. Ie, it > cannot yet update values, and the reading API is fully random-access > by docID (like field cache), not like a posting list, though I > do think we should add an iterator() api (to return flex's DocsEnum) > -- eg I think this would be a good way to track avg doc/field length > for BM25/lnu.ltc scoring. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org