Hi Christoph,
Christoph Kiehl wrote:
I had a look at the search related code during the last days, because we
need better performance for range queries on date fields as well as for
sorting by date fields. These are my thoughts so far:
1. Wouldn't it make sense to exclude the index for the "jcr:system" tree
(which is located at repository/index by default) if the query to
execute doesn't include items from the "jcr:system" tree.
Take for example a query like "my:app//element(*, foo:bar)". This query
only searches for nodes located under "my:app" which excludes nodes from
"jcr:system" and therefore doesn't need to search in the "jcr:system"
index.
I think this is doable. Can you please file a jira issue about this?
As the "jcr:system" might grow quite quickly if you create a lot
versions it might be worth to exclude it.
I'm not sure though how hard it would be to find out if a query needs to
include the "jcr:system" index.
There are two relevant nodes in the query tree to find that out.
- what's the first location step and does it include the jcr:system tree? I
think that's an easy one.
- does the query contain a jcr:deref node? If there is an intermediate result of
a query may dereference into the jcr:system tree.
2. Lucene uses the FieldCaches to speed up sorting and range queries
which is exactly what we are after. Those FieldCaches are per IndexReader.
Jackrabbit uses an IndexSearcher which searches on a single IndexReader
which is most likely to be an instance of CachingMultiReader. So on
every search which builds up a FieldCache this FieldCache instance is
associated with this instance of a CachingMultiReader. On successive
queries which operate on this CachingMultiReader you will get a
tremendous speedup for queries which can reuse those associated
FieldCache instances.
The problem is that Jackrabbit creates a new CachingMultiReader
_everytime_ one of the underlying indexes are modified. This means if
you just change _one_ item in the repository you will need to rebuild
all those FieldCaches because the existing FieldCaches are associated
with the old instance of CachingMultiReader.
This does not only lead to slow search response times for queries which
contains range queries or are sorted by a field but also leads to
massive memory consumption (depending on the size of your indexes)
because there might be multiple instances of CachingMultiReaders in use
if you have a scenario where a lot of queries and item modifications are
executed concurrently.
As far as I understand the solution is to use a MultiSearcher which uses
multiple IndexReaders. Since due to the merging strategy most of the
indexes are stable this means the FieldCaches can be used for a much
longer time.
this is all correct but does not work because. and you actually already found
out why:
I just tried to quickly modify SearchIndex to use a MultiSearcher with
multiple IndexReaders wrapped by IndexSearchers but wasn't successful
because somewhere in DescendantSelfAxisWeight the index readers are
required to implement HierarchyResolver which ReadOnlyIndexReader doesn't.
Using a multi searcher means that you must be able to execute a query on each of
the index segments independently. this is not possible because hierarchy
information is always spread across multiple segments. e.g. a node in one
segment may reference a parent in another segment.
there's also another reason why a multi searcher is not the best solution. it
requires that the fields of a returned FieldDoc contain the values of the
indexed property. If there are lots of values to order the complete set of
values needs to be read into memory. With the current implementation this is not
needed because there is just a single FieldCache that uses integers instead of
the real value. See class SharedFieldSortComparator [1]. the downside of this
approach is that you cannot do a merge sort just using those integers.
a viable solution maybe is a combination of both approaches. use a FieldCache
per index segment (which allows us to cache them for a longer period) but still
use integer values for ordering of nodes within a segment. Then do a merge sort
with a modified SharedFieldSortComparator that reads property values from the
item state manager when nodes are compared across index segments. even though
this requires reading property state, the performance shouldn't suffer too much,
I think. the properties would be read anyway when the query result is iterated,
so it shouldn't harm if they are read already during query execution.
regards
marcel
[1]
https://svn.apache.org/repos/asf/jackrabbit/tags/1.3/jackrabbit-core/src/main/java/org/apache/jackrabbit/core/query/lucene/SharedFieldSortComparator.java
