Hello, I am designing a system with documents having one field containing values such as "Ae1 Br2 Cy8 ...", i.e. a sequence of items made of letters and numbers (max=7 per item), all separated by a space, possibly 200 items per field, with no limit upon the number of documents (although I would not expect more than a few millions documents). The order of these values are important, and I want to search for these, always starting with the first value, and including as many following values as needed: for instance, "Ae1", "Ae1 Br2" would be possible search values.
At first, I indexed these using a space-delimited analyzer, and ran PrefixQueries. I encountered some performance issues though, so ended up building my own tokenizer, which would create tokens for all starting combinations ("Ae1", "Ae1 Br2"...), up to certain limit, called the analysis depth. I would then dynamically create TermQueries to match these tokens when searching under the analysis depth, and PrefixQueries when searching over the analysis depth (the whole string also being indexed as a single token). The performance was great, because TermQueries are very fast, and PrefixQueries are not bad either, when the underlying relevant number of documents is small (which happens to be the case when searching beyond the analysis depth). I have however two questions: one regarding the PrefixQuery, and one regarding the general design. Regarding the PrefixQuery: it seems that it stops matching documents when the length of the searched string exceeds a certain length. Is that the expected behavior, an if so, can I / should I manage this length? Regarding the general design: I have adopted an hybrid approach TermQueries/PrefixQueries, letting clients customize the analysis depth, so as to keep a balance between the performance and the size of the index. I am however not sure this is a good idea: would it be better to tokenize the full string (i.e. analysis depth is infinity, so as to only use TermQueries)? Or could my design be substituted by an altogether different, more successful analysis approach? Thank you in advance for your insights. Kind regards. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org