Notice that while DisjunctionScorer.advance and DisjuntionScorer.advanceAfterCurrent appear to be called in 2.9, in 2.4, I am only seeing DisjuntionScorer.advanceAfterCurrent called.
Can someone explain that? Mark Miller wrote: > Something is very odd about this if they both cover the same search and > the environ for both is identical. Even if one search was done twice, > and we divide the numbers for the new api by 2 - its still *very* odd. > > With 2.4, ScorerDocQueue.topDoc is called half a million times. > With 2.9, its called over 4 million times. > > Huh? > > Thomas Becker wrote: > >> No it's only a single segment. But two calls. One doing a getHitsCount first >> and >> the other doing the actual search. I'll paste both methods below if someone's >> interested. >> >> Will dig into lucene's sources and compare 2.4 search behaviour for my case >> with >> 2.9 tomorrow. It was about time to get more into lucene-core sources anyhow. >> :) >> >> See you tomorrow guys and thanks a lot again! It's a pleasure. >> >> public int getHitsCount(String query, Filter filter) throws >> LuceneServiceException { >> log.debug("getHitsCount('{}, {}')", query, filter); >> if (StringUtils.isBlank(query)) { >> log.warn("getHitsCount: empty lucene query"); >> return 0; >> } >> long startTimeMillis = System.currentTimeMillis(); >> int count = 0; >> >> if (indexSearcher == null) { >> return 0; >> } >> >> BooleanQuery.setMaxClauseCount(MAXCLAUSECOUNT); >> Query q = null; >> try { >> q = createQuery(query); >> TopScoreDocCollector tsdc = >> TopScoreDocCollector.create(1, true); >> indexSearcher.search(q, filter, tsdc); >> count = tsdc.getTotalHits(); >> log.info("getHitsCount: count = {}",count); >> } catch (ParseException ex) { >> throw new LuceneServiceException("invalid lucene >> query:" + query, ex); >> } catch (IOException e) { >> throw new LuceneServiceException(" indexSearcher could >> be corrupted", e); >> } finally { >> long durationMillis = System.currentTimeMillis() - >> startTimeMillis; >> if (durationMillis > slowQueryLimit) { >> log.warn("getHitsCount: Slow query: {} ms, >> query={}", durationMillis, query); >> } >> log.debug("getHitsCount: query took {} ms", >> durationMillis); >> } >> return count; >> } >> >> public List<Document> search(String query, Filter filter, Sort sort, >> int from, >> int size) throws LuceneServiceException { >> log.debug("{} search('{}', {}, {}, {}, {})", new Object[] { >> indexAlias, query, >> filter, sort, from, size }); >> long startTimeMillis = System.currentTimeMillis(); >> >> List<Document> docs = new ArrayList<Document>(); >> if (indexSearcher == null) { >> return docs; >> } >> Query q = null; >> try { >> if (query == null) { >> log.warn("search: lucene query is null..."); >> return docs; >> } >> q = createQuery(query); >> BooleanQuery.setMaxClauseCount(MAXCLAUSECOUNT); >> if (size < 0 || size > maxNumHits) { >> // set hard limit for numHits >> size = maxNumHits; >> if (log.isDebugEnabled()) >> log.debug("search: Size set to >> hardlimit: {} for query: {} with filter: >> {}", new Object[] { size, query, filter }); >> } >> TopFieldCollector collector = >> TopFieldCollector.create(sort, size + from, >> true, false, false, true); >> indexSearcher.search(q, filter, collector); >> if(size > collector.getTotalHits()) >> size = collector.getTotalHits(); >> if (size > 100000) >> log.info("search: size: {} bigger than 100.000 >> for query: {} with filter: >> {}", new Object[] { size, query, filter }); >> TopDocs td = collector.topDocs(from, size); >> ScoreDoc[] scoreDocs = td.scoreDocs; >> for (ScoreDoc scoreDoc : scoreDocs) { >> docs.add(indexSearcher.doc(scoreDoc.doc)); >> } >> } catch (ParseException e) { >> log.warn("search: ParseException: {}", e.getMessage()); >> if (log.isDebugEnabled()) >> log.warn("search: ParseException: ", e); >> return Collections.emptyList(); >> } catch (IOException e) { >> log.warn("search: IOException: ", e); >> return Collections.emptyList(); >> } finally { >> long durationMillis = System.currentTimeMillis() - >> startTimeMillis; >> if (durationMillis > slowQueryLimit) { >> log.warn("search: Slow query: {} ms, query={}, >> indexUsed={}", >> new Object[] { durationMillis, >> query, >> indexSearcher.getIndexReader().directory() }); >> } >> log.debug("search: query took {} ms", durationMillis); >> } >> return docs; >> } >> >> >> Uwe Schindler wrote: >> >> >>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi_mmap_singlereq.png >>>>> >>>>> Have to verify that the last one is not by accident more than one >>>>> >>>>> >>>> request. >>>> >>>> >>>>> Will >>>>> do the run again and then post the required info. >>>>> >>>>> >>>> The last figure shows, that IndexSearcher.searchWithFilter was called >>>> twice >>>> in contrast to the first figure, where IndexSearcher.search was called >>>> only >>>> once. >>>> >>>> >>> I forgot, searchWithFilter it is called per segment in 2.9. If it was only >>> one search, you must have two segments and therefore no optimized index for >>> this to be correct? >>> >>> Uwe >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >>> >> >> > > > -- - Mark http://www.lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org