Re: Solutio for long time highlighting

[David Smiley]

Ah, multi-threaded highlighting.  I implemented that once as a precursor to
ultimately other better things -- the UnifiedHighlighter.

Your ExecutorService ought to be a field on the handler.  In inform() you
can call SolrCore.addCloseHook to ensure this executor is shut down.

I suggest looking at this presentation from a few years ago I did with
Bloomberg at Lucene/Solr Revolution:
https://www.youtube.com/watch?v=tv5qKDKW8kk&t=14s
The UnifiedHighlighter is not enabled by default.  See the documentation:
https://builds.apache.org/job/Solr-reference-guide-master/javadoc/highlighting.html

Still... there is perhaps some value in multi-threading the highlighting
for huge docs, but I think we ultimately found no need after re-engineering
the highlighter.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Aug 28, 2019 at 10:36 AM SOLR4189  wrote:

> Hi all.
>
> In our team we thought about some tricky solution for queries with long
> time
> highlighting. For example, highlighting that takes more than 25 seconds.
> So,
> we created our component that wraps highlighting component of SOLR in this
> way:
>
> public void inform(SolrCore core) {
> . . . .
> subSearchComponent = core.getSearchComponent("highlight");
> . . . .
> }
>
> public void process(ResponseBuilder rb) throws Exception {
> long timeout = 25000;
> ExecutorService exec = null:
> try {
> exec = Executors.newSingleThreadExecutor();
> Future future = exec.submit(() -> {
> try {
> subSearchComponent.process(rb);
> } catch (IOException e) {
> return e;
> }
> return null;
> });
> Exception ex = future.get(timeout, TimeUnit.MILLISECONDS);
> if (ex != null) {
> throw ex;
> }
> } catch ( TimeoutException toe) {
> . . . .
> } catch (Exception e) {
>throw new IOException(e);
> } finally {
> if (exec != null) {
> exec.shutdownNow();
> }
> }
> }
>
> This solution works, but sometime we see that searchers stay open and as a
> result our RAM usage is pretty high (like a memory leak of
> SolrIndexSearcher
> objects). And only after a SOLR service restart they disappear.
>
> What do you think about this solution?
> Maybe exists some built-in function for it?
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Solutio for long time highlighting

[SOLR4189]

Hi all.

In our team we thought about some tricky solution for queries with long time
highlighting. For example, highlighting that takes more than 25 seconds. So,
we created our component that wraps highlighting component of SOLR in this
way:

public void inform(SolrCore core) {
. . . .
subSearchComponent = core.getSearchComponent("highlight");
. . . .
}

public void process(ResponseBuilder rb) throws Exception {
long timeout = 25000;
ExecutorService exec = null:
try {
exec = Executors.newSingleThreadExecutor();
Future future = exec.submit(() -> {
try {
subSearchComponent.process(rb);
} catch (IOException e) {
return e;
} 
return null;
});
Exception ex = future.get(timeout, TimeUnit.MILLISECONDS);
if (ex != null) {
throw ex;
}
} catch ( TimeoutException toe) {
. . . .
} catch (Exception e) {
   throw new IOException(e);
} finally {
if (exec != null) {
exec.shutdownNow();
}
}
}

This solution works, but sometime we see that searchers stay open and as a
result our RAM usage is pretty high (like a memory leak of SolrIndexSearcher
objects). And only after a SOLR service restart they disappear.

What do you think about this solution?
Maybe exists some built-in function for it?



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html