Hi Jukka, Thanks for the reply.
Yes, I am using Jackrabbit 1.2.x and I am not seeing that dramatic of a difference between 1.1.x and the 1.2.x, although I have not done a direct comparison between the two with the same query suite. It looks like adding ordering and or large range queries can significantly impact the query time. I would be very interested in working on and/or looking into optimization strategies and/or experiments. While I can puzzle out the code, and the structure of the query syntax tree, any pointers/documentation would be very much appreciated. In the case of an order by query and large range queries, it looks like significant time is spent in the org.apache.jackrabbit.core.query.lucene.SharedFiledSortComparator class, and that the result set is being walked in order to make sure that each result matches the query parameters and probably for sorting purposes. In the best case it would be good to have a pre-sorted index so that sorted results could be returned much more quickly. Similarly, the ability to define indexes on node properties could be used to speed results. Both of these types of indexes could be modeled and probably implemented similarly to well-known database indexing schemes. Finally, there are also optimizations - more tailored to JSR-170 needs - that could be implemented with Lucene filters - specifically Node Type query constraints - e.g., select * from my:mixin - the Filter could be defined to limit query searches / results to only those nodes that fulfill the specific node type constraint. These filters could be defined for each custom node type defined in the system and pre-calculated. I have not worked with Lucene Filters before, so I am not familiar with their speed - it might be interesting to get input from someone that has more experience with the run time behavior of Filters. Nevertheless, the Lucene docs mention Filters as a method to get around large range queries that inevitably break the 1024 Lucene query term limit. Further information on the Query Syntax Tree that is used by the LuceneQueryBuilder - and "safe" ways to modify it i.e., I would rather only modify the Query Syntax Tree and continue to use the LuceneQueryBuilder for most of the query processing - would be appreciated. -Dave On 3/2/07, Jukka Zitting <[EMAIL PROTECTED]> wrote:
Hi,, On 2/28/07, David Johnson <[EMAIL PROTECTED]> wrote: > "select * from Column where jcr:path like 'Gossip/ColumnName/Columns/%' and > status <> 'hidden' order by publishDate desc" takes 500 ms to execute - this > is just the execution time, I am not actually using or accessing the > NodeIterator. Are you using Jackrabbit 1.2.x? Jackrabbit 1.2 uses lazy loading of query results, which should considerably reduce query execution time by moving the effort to the resulting Node- or RowIterator. In general my rule of thumb so far has been to use the query feature when you want a narrow selection of nodes from a large source set, and to use explicit traversal with filtering when the expected result set includes a considerable percentage of the source set. Optimally the query feature should in all cases be at least equal to traversal speed plus a small constant query parsing and setup overhead. I don't think we are there yet. > Digging into the internals of Jackrabbit, we have noticed that there is an > implementation of RangeQuery that essentially walks the results if the # of > query terms is greater than what Lucene can handle. Reading the Lucene > documentation, it looks like Filters are the recommended method of > implementing "large" range queries, and also seem like a natural for > matching node types - i.e., select * from Column I'm not too familiar with Lucene details to comment on whether Filters would cover everything we need. It would be great if you're interested in pursuing such alternatives! > Is there any ongoing work on query optimization and performance. We would > be very interested in such work, including offering any help that we can. Not apart from the recent lazy loading improvements. Any help would be much appreciated. BR, Jukka Zitting
