Hi,
We are working with a web server and 10 search servers, these 10 servers
have index fragments on it. All available fragments of these search servers
are binding at their start up time. Remote Parallel MultiSearcher is used
for searching on these indices. When a search request comes, first it
lookup(Naming.lookUp), requested fragments is available or not. Then create
a list of Searchable objects for available requested fragments. Doing
further steps of Remote Parallel MultiSearching with this searchables
objects. And the searching works properly with our environment.
But getting the below error once in a while when the search request come,
but there is no error when the requested Index fragment lookup time. Due to
the problem of anyone of the Searcherable fragments, entire search request
goes to exception. Error might be of corrupted Index in a particular
fragments, or machine problem, or remote objects problems.
at first time the error is like this
ERROR [Main Thread] 13 Dec 2007 06:45:41,229 (SearchMaster.java:249) -
caught a class java.rmi.ServerError
with message: Error occurred in server thread; nested exception is:
java.lang.OutOfMemoryError: nativeGetNewTLA
Exception in thread "Main Thread" java.rmi.ServerError: Error occurred in
server thread; nested exception is:
java.lang.OutOfMemoryError: nativeGetNewTLA
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java
:333)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(
TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(
TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(
TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(
StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(
StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at org.apache.lucene.search.RemoteSearchable_Stub.rewrite(Unknown
Source)
at org.apache.lucene.search.MultiSearcher.rewrite(MultiSearcher.java
:261)
at org.apache.lucene.search.ParallelMultiSearcher.rewrite(
ParallelMultiSearcher.java:187)
at org.apache.lucene.search.Query.weight(Query.java:94)
at org.apache.lucene.search.Hits.<init>(Hits.java:49)
at org.apache.lucene.search.Searcher.search(Searcher.java:54)
at com.sumobrain.search.SearchMaster.doSearch(SearchMaster.java:244)
at com.sumobrain.search.SearchMaster.main(SearchMaster.java:672)
Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA
at sun.reflect.ByteVectorImpl.resize(ByteVectorImpl.java:66)
at sun.reflect.ByteVectorImpl.add(ByteVectorImpl.java:45)
at sun.reflect.ClassFileAssembler.emitByte(ClassFileAssembler.java
:56)
at sun.reflect.ClassFileAssembler.emitConstantPoolUTF8(
ClassFileAssembler.java:89)
at sun.reflect.AccessorGenerator.emitCommonConstantPoolEntries(
AccessorGenerator.java:123)
at sun.reflect.MethodAccessorGenerator.generate(
MethodAccessorGenerator.java:333)
at
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(
MethodAccessorGenerator.java:95)
at sun.reflect.ReflectionFactory.newConstructorForSerialization(
ReflectionFactory.java:313)
at java.io.ObjectStreamClass.getSerializableConstructor(
ObjectStreamClass.java:1327)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:437)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
at java.io.ObjectStreamClass.lookup0(ObjectStreamClass.java:310)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java
:547)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java
:1583)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java
:1496)
at java.io.ObjectInputStream.readOrdinaryObject(
ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java
:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java
:290)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(
TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(
TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(
TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
but some times the error message is like this
17 Dec 2007 15:40:59,834 (SearchMaster.java:249) - caught a class
java.lang.NullPointerException
with message: null
Exception in thread "main" java.lang.NullPointerException
at org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(
TermInfosReader.java:100)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:133)
at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:358)
at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:79)
at org.apache.lucene.search.RemoteSearchable.docFreq(
RemoteSearchable.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java
:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(
TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:595)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(
StreamRemoteCall.java:247)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java
:223)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(
RemoteObjectInvocationHandler.java:179)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(
RemoteObjectInvocationHandler.java:132)
at $Proxy0.docFreq(Unknown Source)
at org.apache.lucene.search.MultiSearcher.docFreq(MultiSearcher.java
:140)
at org.apache.lucene.search.ParallelMultiSearcher.docFreq(
ParallelMultiSearcher.java:46)
at org.apache.lucene.search.Similarity.idf(Similarity.java:457)
at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java
:44)
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:145)
at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(
BooleanQuery.java:187)
at org.apache.lucene.search.BooleanQuery$BooleanWeight2.<init>(
BooleanQuery.java:342)
at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java
:384)
at org.apache.lucene.search.Query.weight(Query.java:95)
at org.apache.lucene.search.Hits.<init>(Hits.java:49)
at org.apache.lucene.search.Searcher.search(Searcher.java:54)
at com.sumobrain.search.SearchMaster.doSearch(SearchMaster.java:244)
at com.sumobrain.search.SearchMaster.main(SearchMaster.java:672)
on debugging to the code, error location was at
MultiSearcher : public int docFreq(Term term) throws IOException {
.....
}
in lucene this method is like this
public int docFreq(Term term) throws IOException {
int docFreq = 0;
for (int i = 0; i < searchables.length; i++)
docFreq += searchables[i].docFreq(term);
return docFreq;
}
with this code if anyone of searchebles[i].docFreq[] goes to error,
searching terminated unexpectedly.
Modified the method by removing erroneous Searchable object from Searchables
list, and updated the Searchable list, after removal of object. Modified
Searchables list is used for further search processing.
modified code :
public int docFreq(Term term) throws IOException {
int docFreq = 0;
boolean exceptionOccured = false;
for (int i = 0; i < searchables.length; i++)
if (searchables[i] != null ) {
try {
docFreq += searchables[i].docFreq(term);
} catch (Exception e ) {
// e.printStackTrace();
exceptionOccured = true;
searchables[i] = null;
}
}
if (exceptionOccured) {
List<Searchable> searchableList = new ArrayList<Searchable>();
for (int j = 0; j < searchables.length; j++) {
if (searchables[j] != null ) {
searchableList.add(searchables[j]);
}
}
this.searchables = searchableList.toArray(new Searchable[0]);
}
return docFreq;
}
Modified Searchable list by MultiSearcher is updated with
ParallelMultiSearcher, before starting MultiSearcherThreads.
Modification in ParallelMultiSearcher : TopDocs search(......) method,
updated Searchable list before starting the MultiSearcherThreads
this.searchables = getSearchables(); // get updated Searchable list from
MultiSearcher
With these code modification, we could overcome the above error situation,
and can search with other index fragments without any other error.
Some time we could recover this error by restarting search servers, and can
search into the same fragments without any problem.
Is this problem is due to any lucene error or any of the problem with our
working enviornment.
Could you suggest about this, please.
Thanks
Reeja
