[
https://issues.apache.org/jira/browse/LUCENE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056585#comment-13056585
]
Toke Eskildsen commented on LUCENE-3079:
----------------------------------------
Shai: I completely messed up the JIRA-numbers, clearly I need to go home and
cool my brain. It is fixed now. Sorry for the inconvenience.
Yes, I only created a single root (one dimension) and requested the top-5
facets. You understood the timing measurements correctly. I am sorry that my
table was confusing with regards to memory. The first numbers was the Xmx
required for index build (binary search until I got bored), while the second if
what the JVM reported after the faceting calls were finished. For LUCENE-2369
in the middle test, the faceted search required more memory (aka higher Xmx)
than index build (which could probably have gotten by with even less).
I do not use field cance for LUCENE-2369. It holds a compressed list of
ordinals for tags for the documents in memory, with a few levels of
indirections to handle doublettes. The startup time is basically due to
doublette elimination.
Regarding the memory difference, LUCENE-2369 does not operate at index-time.
This means that is it plain Lucene indexing of terms like hierarchy:a/b/c/d.
Actually I am surprised that it took 128MB for the larger test and I should
probably re-run that with a lower allocation.
My guesstimage, based purely on observation, is that LUCENE-3079 requires heap
relative to the taxonomy size at indexing time. At least with the (assumedly
default) settings I used. Thus the 22M unique values in test #2 is the cause
for the large memory requirement. Looking at the number of unique tags vs.
index memory requirements for case #1 and #2, the factor seems nearly linear.
It seems to fit your recommendation of splitting on large taxonomies?
I'll upload my test class for LUCENE-3079 now. I apologize for its hackish
nature - this was just meant as explorative work.
> Faceting module
> ---------------
>
> Key: LUCENE-3079
> URL: https://issues.apache.org/jira/browse/LUCENE-3079
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/facet
> Reporter: Michael McCandless
> Assignee: Shai Erera
> Fix For: 3.4, 4.0
>
> Attachments: LUCENE-3079-dev-tools.patch, LUCENE-3079.patch,
> LUCENE-3079.patch, LUCENE-3079.patch
>
>
> Faceting is a hugely important feature, available in Solr today but
> not [easily] usable by Lucene-only apps.
> We should fix this, by creating a shared faceting module.
> Ideally, we factor out Solr's faceting impl, and maybe poach/merge
> from other impls (eg Bobo browse).
> Hoss describes some important challenges we'll face in doing this
> (http://markmail.org/message/5w35c2fr4zkiwsz6), copied here:
> {noformat}
> To look at "faceting" as a concrete example, there are big the reasons
> faceting works so well in Solr: Solr has total control over the
> index, knows exactly when the index has changed to rebuild caches, has a
> strict schema so it can make sense of field types and
> pick faceting algos accordingly, has multi-phase distributed search
> approach to get exact counts efficiently across multiple shards, etc...
> (and there are still a lot of additional enhancements and improvements
> that can be made to take even more advantage of knowledge solr has because
> it "owns" the index that we no one has had time to tackle)
> {noformat}
> This is a great list of the things we face in refactoring. It's also
> important because, if Solr needed to be so deeply intertwined with
> caching, schema, etc., other apps that want to facet will have the
> same "needs" and so we really have to address them in creating the
> shared module.
> I think we should get a basic faceting module started, but should not
> cut Solr over at first. We should iterate on the module, fold in
> improvements, etc., and then, once we can fully verify that cutting
> over doesn't hurt Solr (ie lose functionality or performance) we can
> later cutover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
