David Balmain wrote on 07/09/2006 06:44 PM: > On 7/10/06, Chuck Williams <[EMAIL PROTECTED]> wrote: >> Marvin Humphrey wrote on 07/08/2006 11:13 PM: >> > >> > On Jul 8, 2006, at 9:46 AM, Chuck Williams wrote: >> > >> >> Many things would be cleaner in Lucene if fields had a global >> semantics, >> >> i.e., if properties like text vs. binary, Index, Store, >> TermVector, the >> >> appropriate Analyzer, the assignment of Directory in >> ParallelReader (or >> >> ParallelWriter), etc. were a function of just the field name and the >> >> index. >> > >> > In June, Dave Balmain and I discussed the issue extensively on the >> > Ferret list. It might have been nice to use the Lucy list, since a >> > lot of the discussion was about Lucy, but the Lucy lists didn't exist >> > at the time. >> > >> > http://rubyforge.org/pipermail/ferret-talk/2006-June/000536.html >> > >> I think there are a number of problems with that proposal and hope it >> was not adopted. > > Hi Chuck, > > Actually, it was adopted and I'm quite happy with the solution. I'd be > very interested to hear what the number of problems are, besides the > example you've already given. Even if you never use Ferret, it can > only help me improve my software.
Hi David, Thanks for your reply. I'm not aware of other problems beyond the ones I've already cited. After thinking of these, my confidence that there were not others waned. > > I'll start by covering your term-vector example. By adding fixed > index-wide field properties to Ferret I was able to obtain up to a > huge speed improvement during indexing. This is very interesting. Can you say how much? > With the CPU time I gain in Ferret I could > easily re-analyze large fields and build term vectors for them > separately. It's a little more work for less common use cases like > yours but in the end, everyone benifits in terms of performance. Does Ferret work this way, or would that be up to the application? > >> As my earlier example showed, there is at least one >> valid use case where storing a term vector is not an invariant property >> of a field; specifically, when using term vectors to optimize excerpt >> generation, it is best to store them only for fields that have long >> values. This is even a counter-example to Karl's proposal, since a >> single Document may have multiple fields of the same name, some with >> long values and others with short values; multiple fields of the same >> name may legitimately have different TermVector settings even on a >> single Document. > > I think you'll find if you look at the DocumentWriter#writePostings > method that it's "one in, all in" in terms of storing term vectors for > a field. That is, if you have 5 "content" fields and only one of those > is set to store term vectors, then all of the fields will store term > vectors. Right you are, and clearly necessarily so since the values of the multiple fields are implicitly concatenated (with positionIncrementGap). So, Lucene already limits my term vector optimization to the Document level. As it happens, I only use it for large body fields, of which each of my Documents has at most one. > >> I haven't thought of cases where Index or Store would legitimately vary >> across Fields or Documents, but am less convinced there aren't important >> use cases for these as well. Similarly, although it is important to >> allow term vectors to be on or off at the field level, I don't see any >> obvious need to vary the type of term vector (positions, offsets or >> both). > > I think Store could definitely legitimately vary across Fields or > Documents for the same reason your term vectors do. Perhaps you are > indexing pages from the web and you want to cache only the smaller > pages. That's an interesting example, but not as compelling an objection to me (and seemingly not to you either!). The app could always store an empty string without much consequence in this scenario. > >> There are significant benefits to global semantics, as evidenced by the >> fact that several of us independently came to desire this. However, >> deciding what can be global and what cannot is more subtle. > > I agree. I can't see global field semantics making it into Lucene in > the short term. It's a rather large change, particularly if you want > to make full use of the performance benifits it affords. Could you summarize where these derive from? > >> Perhaps the best thing at the Lucene level is to have a notion of >> default semantics for a field name. Whenever a Field of that name is >> constructed, those semantics would be used unless the constructor >> overrides them. This would allow additional constructors on Field with >> simpler signatures for the common case of invariant Field properties. >> It would also allow applications to access the class that holds the >> default field information for an index. The application will know which >> properties it can rely on as invariant and whether or not the set of >> fields is closed. >> >> This approach would preserve upward compatibility and provide, I >> believe, most of the benefits we all seek. >> >> Thoughts? > > If this is all you are going to add, I don't think you'd need to > change Lucene. You could just implement a DocumentFactory in your own > application. Perhaps something like this could go in the contrib > section of Lucene. I've already done it in my application (this weekend). I think Lucene would be better with a mechanism like this built-in as field semantics are usually globally invariant. I'm left wondering whether many of the performance optimizations you've realized might be preserved in a model that allowed selected exceptions, such as the term vector example. > > Also, you mentioned earlier having a field validating query parser. > You can already use > IndexWriter#getFieldNames(IndexReader.FieldOption.INDEXED) to get all > the indexed fields. At least in Lucene, I believe you mean IndexReader.getFieldNames(). However, this is not the same thing. In fact, I submitted bug fixes to ParallelReader a while back (now committed) that were in part due to a similar assumption. The issue is that this method only finds fields that have already been indexed. The model may provide fields that no document in a specific collection has yet used. At least in my application, this distinction is important. I have a common model used to build many indexes, with search and indexing performed simultaneously. At any point in time in any given collection, a field available in the model may or may not have occurred. Queries need to be validated against the model, not against the specific collection. Thanks again, Chuck --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
