Hi, We are working with a web server and 10 search servers, these 10 servers have index fragments on it. All available fragments of these search servers are binding at their start up time. Remote Parallel MultiSearcher is used for searching on these indices. When a search request comes, first it lookup(Naming.lookUp), requested fragments is available or not. Then create a list of Searchable objects for available requested fragments. Doing further steps of Remote Parallel MultiSearching with this searchables objects. And the searching works properly with our environment.
But getting the below error once in a while when the search request come, but there is no error when the requested Index fragment lookup time. Due to the problem of anyone of the Searcherable fragments, entire search request goes to exception. Error might be of corrupted Index in a particular fragments, or machine problem, or remote objects problems. at first time the error is like this ERROR [Main Thread] 13 Dec 2007 06:45:41,229 (SearchMaster.java:249) - caught a class java.rmi.ServerError with message: Error occurred in server thread; nested exception is: java.lang.OutOfMemoryError: nativeGetNewTLA Exception in thread "Main Thread" java.rmi.ServerError: Error occurred in server thread; nested exception is: java.lang.OutOfMemoryError: nativeGetNewTLA at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java :333) at sun.rmi.transport.Transport$1.run(Transport.java:159) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages( TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0( TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run( TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer( StreamRemoteCall.java:255) at sun.rmi.transport.StreamRemoteCall.executeCall( StreamRemoteCall.java:233) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142) at org.apache.lucene.search.RemoteSearchable_Stub.rewrite(Unknown Source) at org.apache.lucene.search.MultiSearcher.rewrite(MultiSearcher.java :261) at org.apache.lucene.search.ParallelMultiSearcher.rewrite( ParallelMultiSearcher.java:187) at org.apache.lucene.search.Query.weight(Query.java:94) at org.apache.lucene.search.Hits.<init>(Hits.java:49) at org.apache.lucene.search.Searcher.search(Searcher.java:54) at com.sumobrain.search.SearchMaster.doSearch(SearchMaster.java:244) at com.sumobrain.search.SearchMaster.main(SearchMaster.java:672) Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA at sun.reflect.ByteVectorImpl.resize(ByteVectorImpl.java:66) at sun.reflect.ByteVectorImpl.add(ByteVectorImpl.java:45) at sun.reflect.ClassFileAssembler.emitByte(ClassFileAssembler.java :56) at sun.reflect.ClassFileAssembler.emitConstantPoolUTF8( ClassFileAssembler.java:89) at sun.reflect.AccessorGenerator.emitCommonConstantPoolEntries( AccessorGenerator.java:123) at sun.reflect.MethodAccessorGenerator.generate( MethodAccessorGenerator.java:333) at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor( MethodAccessorGenerator.java:95) at sun.reflect.ReflectionFactory.newConstructorForSerialization( ReflectionFactory.java:313) at java.io.ObjectStreamClass.getSerializableConstructor( ObjectStreamClass.java:1327) at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:437) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413) at java.io.ObjectStreamClass.lookup0(ObjectStreamClass.java:310) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java :547) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java :1583) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java :1496) at java.io.ObjectInputStream.readOrdinaryObject( ObjectInputStream.java:1732) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java :1329) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java :290) at sun.rmi.transport.Transport$1.run(Transport.java:159) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages( TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0( TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run( TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) but some times the error message is like this 17 Dec 2007 15:40:59,834 (SearchMaster.java:249) - caught a class java.lang.NullPointerException with message: null Exception in thread "main" java.lang.NullPointerException at org.apache.lucene.index.TermInfosReader.ensureIndexIsRead( TermInfosReader.java:100) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:133) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:358) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:79) at org.apache.lucene.search.RemoteSearchable.docFreq( RemoteSearchable.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) at sun.rmi.transport.Transport$1.run(Transport.java:153) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:149) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java :460) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run( TCPTransport.java:701) at java.lang.Thread.run(Thread.java:595) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer( StreamRemoteCall.java:247) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java :223) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod( RemoteObjectInvocationHandler.java:179) at java.rmi.server.RemoteObjectInvocationHandler.invoke( RemoteObjectInvocationHandler.java:132) at $Proxy0.docFreq(Unknown Source) at org.apache.lucene.search.MultiSearcher.docFreq(MultiSearcher.java :140) at org.apache.lucene.search.ParallelMultiSearcher.docFreq( ParallelMultiSearcher.java:46) at org.apache.lucene.search.Similarity.idf(Similarity.java:457) at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java :44) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:145) at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>( BooleanQuery.java:187) at org.apache.lucene.search.BooleanQuery$BooleanWeight2.<init>( BooleanQuery.java:342) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java :384) at org.apache.lucene.search.Query.weight(Query.java:95) at org.apache.lucene.search.Hits.<init>(Hits.java:49) at org.apache.lucene.search.Searcher.search(Searcher.java:54) at com.sumobrain.search.SearchMaster.doSearch(SearchMaster.java:244) at com.sumobrain.search.SearchMaster.main(SearchMaster.java:672) on debugging to the code, error location was at MultiSearcher : public int docFreq(Term term) throws IOException { ..... } in lucene this method is like this public int docFreq(Term term) throws IOException { int docFreq = 0; for (int i = 0; i < searchables.length; i++) docFreq += searchables[i].docFreq(term); return docFreq; } with this code if anyone of searchebles[i].docFreq[] goes to error, searching terminated unexpectedly. Modified the method by removing erroneous Searchable object from Searchables list, and updated the Searchable list, after removal of object. Modified Searchables list is used for further search processing. modified code : public int docFreq(Term term) throws IOException { int docFreq = 0; boolean exceptionOccured = false; for (int i = 0; i < searchables.length; i++) if (searchables[i] != null ) { try { docFreq += searchables[i].docFreq(term); } catch (Exception e ) { // e.printStackTrace(); exceptionOccured = true; searchables[i] = null; } } if (exceptionOccured) { List<Searchable> searchableList = new ArrayList<Searchable>(); for (int j = 0; j < searchables.length; j++) { if (searchables[j] != null ) { searchableList.add(searchables[j]); } } this.searchables = searchableList.toArray(new Searchable[0]); } return docFreq; } Modified Searchable list by MultiSearcher is updated with ParallelMultiSearcher, before starting MultiSearcherThreads. Modification in ParallelMultiSearcher : TopDocs search(......) method, updated Searchable list before starting the MultiSearcherThreads this.searchables = getSearchables(); // get updated Searchable list from MultiSearcher With these code modification, we could overcome the above error situation, and can search with other index fragments without any other error. Some time we could recover this error by restarting search servers, and can search into the same fragments without any problem. Is this problem is due to any lucene error or any of the problem with our working enviornment. Could you suggest about this, please. Thanks Reeja