[
https://issues.apache.org/jira/browse/LUCENE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12690100#action_12690100
]
Ethan Tao commented on LUCENE-1304:
-----------------------------------
Thank you !!
> Memory Leak when using Custom Sort (i.e., DistanceSortSource) of LocalLucene
> with Lucene
> ----------------------------------------------------------------------------------------
>
> Key: LUCENE-1304
> URL: https://issues.apache.org/jira/browse/LUCENE-1304
> Project: Lucene - Java
> Issue Type: Bug
> Components: Search
> Affects Versions: 2.3
> Environment: Windows/JDK 1.6
> Reporter: Ethan Tao
> Fix For: 2.9
>
> Attachments: LUCENE-1304.patch
>
>
> We had the memory leak issue when using DistanceSortSource of LocalLucene for
> repeated query/search. In about 450 queries, we are experiencing out of
> memory error. After dig in the code, we found the problem source is coming
> from Lucene package, the way how it handles "custom" type comparator. Lucene
> internally caches all created comparators. In the case of query using
> LocalLucene, we create new comparator for every search due to different
> lon/lat and query terms. This causes major memory leak as the cached
> comparators are also holding memory for other large objects (e.g., bit sets).
> The solution we came up with: ( the proposed change from Lucene is 1 and 3
> below)
> 1. In Lucene package, create new file SortComparatorSourceUncacheable.java:
> package org.apache.lucene.search;
> import org.apache.lucene.index.IndexReader;
> import java.io.IOException;
> import java.io.Serializable;
> public interface SortComparatorSourceUncacheable extends Serializable {
> }
> 2. Have your custom sort class to implement the interface
> public class LocalSortSource extends DistanceSortSource implements
> SortComparatorSourceUncacheable {
> ...
> }
> 3. Modify Lucene's FieldSorterHitQueue.java to bypass caching for custom
> sort comparator:
> Index: FieldSortedHitQueue.java
> ===================================================================
> --- FieldSortedHitQueue.java (revision 654583)
> +++ FieldSortedHitQueue.java (working copy)
> @@ -53,7 +53,12 @@
> this.fields = new SortField[n];
> for (int i=0; i<n; ++i) {
> String fieldname = fields[i].getField();
> - comparators[i] = getCachedComparator (reader, fieldname,
> fields[i].getType(), fields[i].getLocale(), fields[i].getFactory());
> +
> + if(fields[i].getFactory() instanceof SortComparatorSourceUncacheable)
> { // no caching to avoid memory leak
> + comparators[i] = getComparator (reader, fieldname,
> fields[i].getType(), fields[i].getLocale(), fields[i].getFactory());
> + } else {
> + comparators[i] = getCachedComparator (reader, fieldname,
> fields[i].getType(), fields[i].getLocale(), fields[i].getFactory());
> + }
>
> if (comparators[i].sortType() == SortField.STRING) {
> this.fields[i] = new SortField (fieldname,
> fields[i].getLocale(), fields[i].getReverse());
> @@ -157,7 +162,18 @@
> SortField[] getFields() {
> return fields;
> }
> -
> +
> + static ScoreDocComparator getComparator (IndexReader reader, String field,
> int type, Locale locale, SortComparatorSource factory)
> + throws IOException {
> + if (type == SortField.DOC) return ScoreDocComparator.INDEXORDER;
> + if (type == SortField.SCORE) return ScoreDocComparator.RELEVANCE;
> + FieldCacheImpl.Entry entry = (factory != null)
> + ? new FieldCacheImpl.Entry (field, factory)
> + : new FieldCacheImpl.Entry (field, type, locale);
> + return (ScoreDocComparator)Comparators.createValue(reader, entry);
> + }
> +
> +
> Otis suggests that I put this in Jira. I 'll attach a patch shortly for
> review.
> -Ethan
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
