I tried it with just Lucene + RMI, and that works just fine. It's actually
based on the Lucene In Action e-book topic on how to use
ParallelMultiSearcher (chap.5). The relevant code snippet follows:
/*
* search server:
* This is the code frag for the search server, which enters
* a wait-loop to accept requests on port 1099.
* This server implementation is run on 2+ separate boxes, one
* is a "master" while the rest are as "slaves", where master is
* the main entry point which searches both it's local indexes,
* and sends requests to each slave, which only searches its own
* local indexes and reports back results to the master.
*/
//private Vector<Searchable> _searchables;
//private Vector<String> _localDirs;
// ...
// add local dirs as searchables..
for (int i=0; i<_localDirs.size(); i++) {
System.out.println("local searchable: " + _localDirs.get(i) + " ..");
_searchables.add(new IndexSearcher(_localDirs.get(i)));
}
// add remote nodes (slaves) as searchables..
// note: only master will do this, the slaves only looks at its local
indexes..
if (_remoteNodes != null) {
Collection nodes = _remoteNodes.values();
Iterator it = nodes.iterator();
String node = "";
while (it.hasNext()) {
node = (String) it.next();
try {
// remote nodes (slaves) also reachable via port 1099
_searchables.add((Searchable) Naming.lookup("//" + node +
":1099/" + _DEFAULT_SVC_NAME_));
System.out.println("remote searchable: " + node + " ..");
} catch (java.rmi.ConnectException e) {
System.err.println("ERROR: unable to connect to node=" + node + "
...");
}
}
}
// just some glue to prepare list of searchables for ParallelMultiSearcher
constructor..
Searchable[] sch = new Searchable[_searchables.size()];
for (int i=0; i<_searchables.size(); i++) {
sch[i] = _searchables.get(i);
}
// start up server..
System.setSecurityManager(new RMISecurityManager());
LocateRegistry.createRegistry(_port);
Searcher parallelSearcher = new ParallelMultiSearcher(sch);
RemoteSearchable parallelImpl = new RemoteSearchable(parallelSearcher);
Naming.rebind("//" + _nodeID + ":" + _port + "/" + _DEFAULT_SVC_NAME_,
parallelImpl);
System.out.println("SearchServer started " +
"(nodeID=" + _nodeID +
", port=" + _port +
", role=" + ((_remoteNodes!=null)?"master":"slave") +
", # searchables=" + _searchables.size() + ")...");
// enters wait state, ready to accept requests on port 1099...
========================
/*
* search client
* This basically does an RMI naming lookup to get a reference to
* the master node on port 1099, then sends a search query..
*/
TermQuery query = new TermQuery(new Term("body", word));
MultiSearcher searcher = new MultiSearcher(new
Searchable[]{_lookupRemote(_DEFAULT_SVC_NAME_)});
Hits hits = searcher.search(query);
Document doc = null;
for (int i=0; i<hits.length(); i++) {
doc = hits.doc(i);
// able to get hit info here...
}
// .....
private Searchable _lookupRemote(String svcName) throws Exception {
return (Searchable) Naming.lookup("//" + _host + ":" + _port + "/" +
svcName);
}
========================
From both of the above code, I am able to start a server on box1 (master),
another server on box2 (slave), then invoke a client that queries box1,
which can get results from searching indexes in box1+box2. With this
working, that's when I tried to incorporate ParallelMultiSearcher on Solr's
SolrIndexSearcher, since I saw that it is the place where it uses Lucene's
IndexSearcher. I replaced it with ParallelMultiSearcher, where it is
initialized similar to the client code I mentioned above.
From that, it seems like Solr itself needs to marshall and unmarshall the
searcher instance SolrIndexSearcher holds, and because the
ParallelMultiSearcher is initialized with RMI stubs, it fails to proceed
with such marshall/unmarshall internal actions. As mentioned in the first
email, if I use ParallelMultiSearcher to only look at local indexes (no RMI
stub), Solr works just fine. So I'm wondering if there is a way use
SolrIndexSearcher to search both local and remote indexes, even if not
through the RMI solution Lucene's ebook has suggested via its
ParallelMultiSearcher class.
tia,
Koji
On 5/9/06, Chris Hostetter <[EMAIL PROTECTED]> wrote:
I don't really know a lot about RMI, but as i understand it, Serialization
is a core neccessity -- if the arguments you want to pass to your Remote
Method aren't serializable, then RMI can't pass those argument across the
wire.
That said: it's not clear to me from the psuedocode/stacktrace you
included *what* isn't serializable ... is it a Solr class or a core Lucene
class?
If it's a Lucene class, you may want to start by making a small proof
of concept RMI app that just uses the Lucene core classes, once that
works then try your changes in Solr.
: Date: Tue, 9 May 2006 02:32:45 -0700
: From: Koji Miyamoto <[EMAIL PROTECTED]>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: extending SolrIndexSearcher
:
: Hi,
:
: I am looking at extending the source code for SolrIndexSearcher for my
own
: purposes. Basically, I am trying to replace the use of Lucene's
: IndexSearcher with a ParallelMultiSearcher version so that I can have a
: query search both locally available indexes as well as remote indexes
: available only via RMI. This ParallelMultiSearcher is instantiated to
: consist of both local and remote Searchable references. The local
: Searchables are simply IndexSearcher instances tied to local disk
(separate
: indexes), while the remote Searchables are made reachable via RMI.
:
: In essence, where it used to be:
:
: IndexSearcher searcher = new IndexSearcher(reader);
:
: it is now: (not the actual code but similar)
:
: Searchable[] searchables = new Searchable[3];
: for (int i=0; i<2; i++) {
: // Local searchable:
: searchables[i] = new IndexSearcher("/disk" + i + "/index");
: }
:
: // RMI searchable: throws exception during search..
: searchables[2] = (Searchable) Naming.lookup
: ("//remote_host:1099/remote_svc");
:
: ParallelMultiSearcher searcher = new ParallelMultiSearcher(sch);
:
: When I build the source and use it (the short story, by replacing the
: relevant class file(s) within solr.war used by the example jetty
: implementation), it starts up just fine. If I comment out the RMI
: searchable line, submission of a search query to Jetty/Solr works just
fine,
: and it is able to search any number of indexes. However, with the RMI
: searchable uncommented out, I get an exception thrown (here's the ending
of
: it):
:
: May 9, 2006 1:38:07 AM org.apache.solr.core.SolrException log
: SEVERE: java.rmi.MarshalException: error marshalling arguments; nested
: exception is:
: java.io.NotSerializableException:
: org.apache.lucene.search.MultiSearcher$1
: at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:122)
: at org.apache.lucene.search.RemoteSearchable_Stub.search(Unknown
: Source)
: at org.apache.lucene.search.MultiSearcher.search(
MultiSearcher.java
: :248)
: at org.apache.lucene.search.Searcher.search(Searcher.java:116)
: at org.apache.lucene.search.Searcher.search(Searcher.java:95)
: at org.apache.solr.search.SolrIndexSearcher.getDocListNC(
: SolrIndexSearcher.java:794)
: at org.apache.solr.search.SolrIndexSearcher.getDocListC(
: SolrIndexSearcher.java:712)
: at org.apache.solr.search.SolrIndexSearcher.getDocList(
: SolrIndexSearcher.java:605)
: at org.apache.solr.request.StandardRequestHandler.handleRequest(
: StandardRequestHandler.java:106)
:
: So it looks like it requires Serialization somehow to get it to work.
: Wondering if anyone has any ideas to get around this problem.
:
: tia,
: Koji
:
-Hoss
