This would work a lot better if you did the join at index time. For
each paper, add a field with all the related drug names (or whatever
you want to search for), then search on that field.
With the current design, it will never be fast and never scale. Each
lookup has a cost, so expanding a query to a thousand terms will
always be slow. Distributing the query to multiple shards will only
make a bad design slightly faster.
This is fundamental to search index design. The schema is flat, fully-
denormalized, no joins. You tag each document with the terms that you
will use to find it. Then you search for those terms directly.
wunder
On Sep 25, 2009, at 7:52 AM, Luo, Jeff wrote:
We are searching strings, not numbers. The reason we are doing this
kind
of query is that we have two big indexes, say, a collection of
medicine
drugs and a collection of research papers. I first run a query against
the drugs index and get 102400 unique drug names back. Then I need to
find all the research papers where one or more of the 102400 drug
names
are mentioned, hence the large OR query. This is a kind of JOIN query
between 2 indexes, which an article in the lucid web site comparing
databases and search engines briefly touched.
I was able to issue 100 parallel small queries against solr shards and
get the results back successfully (even sorted). My custom code is
less
than 100 lines, mostly in my SearchHandler.handleRequestBody. But I
have
problem summing up the correct facet counts because the faceting
counts
from each shard are not disjunctive.
Based on what is suggested by two other responses to my question, I
think it is possible that the master can pass the original large query
to each shard, and each shard will split the large query into 100
lower
level disjunctive lucene queries, fire them against its Lucene index
in
a parallel way and merge the results. Then each shard shall only
return
1(instead of 100) result set to the master with disjunctive faceting
counts. It seems that the faceting problem can be solved in this
way. I
would appreciate it if you could let me know if this approach is
feasible and correct; what solr plug-ins are needed(my guess is a
custom
parser and query-component?)
Thanks,
Jeff
-----Original Message-----
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Thursday, September 24, 2009 10:01 AM
To: solr-dev@lucene.apache.org
Subject: [PMX:FAKE_SENDER] Re: large OR-boolean query
On Sep 23, 2009, at 4:26 PM, Luo, Jeff wrote:
Hi,
We are experimenting a parallel approach to issue a large OR-Boolean
query, e.g., keywords:(1 OR 2 OR 3 OR ... OR 102400), against several
solr shards.
The way we are trying is to break the large query into smaller ones,
e.g.,
the example above can be broken into 10 small queries: keywords:(1
OR 2
OR 3 OR ... OR 1024), keywords:(1025 OR 1026 OR 1027 OR ... OR 2048),
etc
Now each shard will get 10 requests and the master will merge the
results coming back from each shard, similar to the regular
distributed
search.
Can you tell us a little bit more about the why/what of this? Are you
really searching numbers or are those just for example? Do you care
about the score or do you just need to know whether the result is
there or not?
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
