We conducted a test on our search for 500 requests
given in 27 seconds. We noticed that in the first 5
seconds, the results were coming in 100 to 500 ms. But
as the queue size kept increasing, the response time
of the search increased drastically to approx 80-100
seconds.
Please find enclosed jvmdump.txt which contains a dump
of our search program after about 20 seconds of
starting the program.
Also enclosed is the file queries.txt which contains
few sample search queries.
Please note that this is done on a sample of 400,000
documents (450MB) on P4 having 1GB RAM.
Kindly let us know if this helps to identify the cause
of slow response.
Jayant
--- Doug Cutting <[EMAIL PROTECTED]> wrote: > Jayant
Kumar wrote:
> > We recently tested lucene with an index size of 2
> GB
> > which has about 1,500,000 documents, each document
> > having about 25 fields. The frequency of search
> was
> > about 20 queries per second. This resulted in an
> > average response time of about 20 seconds approx
> > per search.
>
> That sounds slow, unless your queries are very
> complex. What are your
> queries like?
>
> > What we observed was that lucene queues
> > the queries and does not release them until the
> > results are found. so the queries that have come
> in
> > later take up about 500 seconds. Please let us
> know
> > whether there is a technique to optimize lucene in
> > such circumstances.
>
> Multiple queries executed from different threads
> using a single searcher
> should not queue, but should run in parallel. A
> technique to find out
> where threads are queueing is to get a thread dump
> and see where all of
> the threads are stuck. In Solaris and Linux,
> sending the JVM a SIGQUIT
> will give a thread dump. On Windows, use
> Control-Break.
>
> Doug
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [EMAIL PROTECTED]
> For additional commands, e-mail:
> [EMAIL PROTECTED]
>
________________________________________________________________________
Yahoo! India Matrimony: Find your partner online.
http://yahoo.shaadi.com/india-matrimony/
"Thread-14" prio=1 tid=0x080a7420 nid=0x468e waiting for monitor entry
[4d61a000..4d61ac18]
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112)
- waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader)
at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51)
at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
at org.apache.lucene.search.Hits.<init>(Hits.java:43)
at org.apache.lucene.search.Searcher.search(Searcher.java:33)
at org.apache.lucene.search.Searcher.search(Searcher.java:27)
at resdex.searchinc.getHits(searchinc.java:752)
at resdex.searchinc.Search(searchinc.java:943)
at resdex.searchinctest.conductTestSearch(searchinctest.java:99)
at resdex.Server$Handler.run(Server.java:64)
at java.lang.Thread.run(Thread.java:534)
"Thread-12" prio=1 tid=0x080a58e0 nid=0x468e waiting for monitor entry
[4d51a000..4d51ad18]
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112)
- waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader)
at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51)
at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
at org.apache.lucene.search.Hits.<init>(Hits.java:43)
at org.apache.lucene.search.Searcher.search(Searcher.java:33)
at org.apache.lucene.search.Searcher.search(Searcher.java:27)
at resdex.searchinc.getHits(searchinc.java:752)
at resdex.searchinc.Search(searchinc.java:943)
at resdex.searchinctest.conductTestSearch(searchinctest.java:99)
at resdex.Server$Handler.run(Server.java:64)
at java.lang.Thread.run(Thread.java:534)
"Thread-11" prio=1 tid=0x080a4b40 nid=0x468e waiting for monitor entry
[4d49a000..4d49ad98]
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112)
- waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader)
at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51)
at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
at org.apache.lucene.search.Hits.<init>(Hits.java:43)
at org.apache.lucene.search.Searcher.search(Searcher.java:33)
at org.apache.lucene.search.Searcher.search(Searcher.java:27)
at resdex.searchinc.getHits(searchinc.java:752)
at resdex.searchinc.Search(searchinc.java:943)
at resdex.searchinctest.conductTestSearch(searchinctest.java:99)
at resdex.Server$Handler.run(Server.java:64)
at java.lang.Thread.run(Thread.java:534)
"Thread-8" prio=1 tid=0x080a2260 nid=0x468e waiting for monitor entry
[4d31a000..4d31af18]
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:112)
- waiting to lock <0x44c95228> (a org.apache.lucene.index.TermInfosReader)
at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51)
at org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:164)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
at org.apache.lucene.search.Hits.<init>(Hits.java:43)
at org.apache.lucene.search.Searcher.search(Searcher.java:33)
at org.apache.lucene.search.Searcher.search(Searcher.java:27)
at resdex.searchinc.getHits(searchinc.java:752)
at resdex.searchinc.Search(searchinc.java:943)
at resdex.searchinctest.conductTestSearch(searchinctest.java:99)
at resdex.Server$Handler.run(Server.java:64)
at java.lang.Thread.run(Thread.java:534)
"DestroyJavaVM" prio=1 tid=0x08052140 nid=0x468e waiting on condition [0..bfffd004]
"Thread-0" prio=1 tid=0x0820cdb0 nid=0x468e runnable [4cf0a000..4cf0a318]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x44ef91e0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at resdex.Server.run(Server.java:95)
at java.lang.Thread.run(Thread.java:534)
"Signal Dispatcher" daemon prio=1 tid=0x0809ce70 nid=0x468e waiting on condition [0..0]
"Finalizer" daemon prio=1 tid=0x080884d8 nid=0x468e in Object.wait()
[4c921000..4c921598]
at java.lang.Object.wait(Native Method)
- waiting on <0x44c29be0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x44c29be0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x080878e8 nid=0x468e in Object.wait()
[4c8a1000..4c8a1618]
at java.lang.Object.wait(Native Method)
- waiting on <0x44c29c48> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:429)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
- locked <0x44c29c48> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=1 tid=0x080865c8 nid=0x468e runnable
"VM Periodic Task Thread" prio=1 tid=0x0809f680 nid=0x468e waiting on condition
"Suspend Checker Thread" prio=1 tid=0x0809c4b8 nid=0x468e runnable
Query : ( ( ( ( ( FIELD1: proof OR FIELD2: proof OR FIELD3: proof OR FIELD4:
proof OR FIELD5: proof OR FIELD6: proof OR FIELD7: proof ) AND ( FIELD1: "george
bush" OR FIELD2: "george bush" OR FIELD3: "george bush" OR FIELD4: "george bush" OR
FIELD5: "george bush" OR FIELD6: "george bush" OR FIELD7: "george bush" ) ) AND (
FIELD1: script OR FIELD2: script OR FIELD3: script OR FIELD4: script OR FIELD5:
script OR FIELD6: script OR FIELD7: script ) ) AND ( ( FIELD1: san OR FIELD2:
san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san )
OR ( ( FIELD1: war OR FIELD2: war OR FIELD3: war OR FIELD4: war OR FIELD5: war
OR FIELD6: war OR FIELD7: war ) OR ( ( FIELD1: gulf OR FIELD2: gulf OR FIELD3:
gulf OR FIELD4: gulf OR FIELD5: gulf OR FIELD6: gulf OR FIELD7: gulf ) OR ( (
FIELD1: laden OR FIELD2: laden OR FIELD3: laden OR FIELD4: laden OR FIELD5: laden
OR FIELD6: laden OR FIELD7: laden ) OR ( ( FIELD1: ttouristeat OR FIELD2:
ttouristeat OR FIELD3: ttouristeat OR FIELD4: ttouristeat OR FIELD5: ttouristeat OR
FIELD6: ttouristeat OR FIELD7: ttouristeat ) OR ( ( FIELD1: pow OR FIELD2: pow OR
FIELD3: pow OR FIELD4: pow OR FIELD5: pow OR FIELD6: pow OR FIELD7: pow ) OR (
FIELD1: bin OR FIELD2: bin OR FIELD3: bin OR FIELD4: bin OR FIELD5: bin OR
FIELD6: bin OR FIELD7: bin ) ) ) ) ) ) ) ) ) AND RANGE: ([ 0800 TO 1100 ])
AND ( S_IDa: (7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14 OR 15 OR 16 OR 17 ) or
S_IDb: (2 ) )
Time taken : 86304 ms
Query : ( ( ( FIELD1: india OR FIELD2: india OR FIELD3: india OR FIELD4: india
OR FIELD5: india OR FIELD6: india OR FIELD7: india ) AND ( ( FIELD1: tourist OR
FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR
FIELD6: tourist OR FIELD7: tourist ) OR ( ( FIELD1: attraction OR FIELD2:
attraction OR FIELD3: attraction OR FIELD4: attraction OR FIELD5: attraction OR
FIELD6: attraction OR FIELD7: attraction ) OR ( ( FIELD1: ps OR FIELD2: ps OR
FIELD3: ps OR FIELD4: ps OR FIELD5: ps OR FIELD6: ps OR FIELD7: ps ) OR ( (
FIELD1: pm OR FIELD2: pm OR FIELD3: pm OR FIELD4: pm OR FIELD5: pm OR FIELD6: pm
OR FIELD7: pm ) OR ( ( FIELD1: fico OR FIELD2: fico OR FIELD3: fico OR FIELD4:
fico OR FIELD5: fico OR FIELD6: fico OR FIELD7: fico ) OR ( ( FIELD1: basis OR
FIELD2: basis OR FIELD3: basis OR FIELD4: basis OR FIELD5: basis OR FIELD6: basis
OR FIELD7: basis ) OR ( ( FIELD1: abap OR FIELD2: abap OR FIELD3: abap OR
FIELD4: abap OR FIELD5: abap OR FIELD6: abap OR FIELD7: abap ) OR ( FIELD1:
destination OR FIELD2: destination OR FIELD3: destination OR FIELD4: destination OR
FIELD5: destination OR FIELD6: destination OR FIELD7: destination ) ) ) ) ) )
) ) ) ) AND LOC_A: (27 )
Time taken : 83067 ms
Query : ( ( ( ( ( ( ( ( ( FIELD1: usa OR FIELD2: usa OR FIELD3: usa OR
FIELD4: usa OR FIELD5: usa OR FIELD6: usa OR FIELD7: usa ) OR ( FIELD1: "vegas /
chicago" OR FIELD2: "vegas / chicago" OR FIELD3: "vegas / chicago" OR FIELD4:
"vegas / chicago" OR FIELD5: "vegas / chicago" OR FIELD6: "vegas / chicago" OR
FIELD7: "vegas / chicago" ) ) AND ( ( FIELD1: " & " OR FIELD2: " & " OR FIELD3: "
& " OR FIELD4: " & " OR FIELD5: " & " OR FIELD6: " & " OR FIELD7: " & " ) AND ( (
FIELD1: fun OR FIELD2: fun OR FIELD3: fun OR FIELD4: fun OR FIELD5: fun OR
FIELD6: fun OR FIELD7: fun ) AND ( FIELD1: large OR FIELD2: large OR FIELD3: large
OR FIELD4: large OR FIELD5: large OR FIELD6: large OR FIELD7: large ) ) ) ) AND
( ( FIELD1: fc OR FIELD2: fc OR FIELD3: fc OR FIELD4: fc OR FIELD5: fc OR
FIELD6: fc OR FIELD7: fc ) AND ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR
FIELD4: san OR FIELD5: san OR FIELD6: san OR FIELD7: san ) AND ( ( FIELD1:
tourist OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5:
tourist OR FIELD6: tourist OR FIELD7: tourist ) AND ( FIELD1: casino OR FIELD2:
casino OR FIELD3: casino OR FIELD4: casino OR FIELD5: casino OR FIELD6: casino OR
FIELD7: casino ) ) ) ) ) AND ( ( FIELD1: fc OR FIELD2: fc OR FIELD3: fc OR
FIELD4: fc OR FIELD5: fc OR FIELD6: fc OR FIELD7: fc ) OR ( ( FIELD1: san OR
FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san OR FIELD6: san OR
FIELD7: san ) OR ( ( FIELD1: tourist OR FIELD2: tourist OR FIELD3: tourist OR
FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR FIELD7: tourist ) OR (
FIELD1: casino OR FIELD2: casino OR FIELD3: casino OR FIELD4: casino OR FIELD5:
casino OR FIELD6: casino OR FIELD7: casino ) ) ) ) ) AND ( ( FIELD1: fc OR
FIELD2: fc OR FIELD3: fc OR FIELD4: fc OR FIELD5: fc OR FIELD6: fc OR FIELD7: fc
) OR ( ( FIELD1: san OR FIELD2: san OR FIELD3: san OR FIELD4: san OR FIELD5: san
OR FIELD6: san OR FIELD7: san ) OR ( ( FIELD1: tourist OR FIELD2: tourist OR
FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR FIELD6: tourist OR
FIELD7: tourist ) OR ( FIELD1: casino OR FIELD2: casino OR FIELD3: casino OR
FIELD4: casino OR FIELD5: casino OR FIELD6: casino OR FIELD7: casino ) ) ) ) )
AND ( FIELD1: war OR FIELD2: war OR FIELD3: war OR FIELD4: war OR FIELD5: war OR
FIELD6: war OR FIELD7: war ) ) AND ( FIELD1: war OR FIELD2: war OR FIELD3: war OR
FIELD4: war OR FIELD5: war OR FIELD6: war OR FIELD7: war ) ) ) AND RANGE: ([
0000 TO 0100 ]) AND S_IDa: (7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14 OR 15 OR 16 OR
17 )
Time taken : 86874 ms
Query : ( ( ( ( FIELD1: india OR FIELD2: india OR FIELD3: india OR FIELD4:
india OR FIELD5: india OR FIELD6: india OR FIELD7: india ) AND ( ( FIELD1:
touristms OR FIELD2: touristms OR FIELD3: touristms OR FIELD4: touristms OR
FIELD5: touristms OR FIELD6: touristms OR FIELD7: touristms ) OR ( ( FIELD1:
touristis OR FIELD2: touristis OR FIELD3: touristis OR FIELD4: touristis OR
FIELD5: touristis OR FIELD6: touristis OR FIELD7: touristis ) OR ( FIELD1: tourist
OR FIELD2: tourist OR FIELD3: tourist OR FIELD4: tourist OR FIELD5: tourist OR
FIELD6: tourist OR FIELD7: tourist ) ) ) ) AND ( ( FIELD1: h1 OR FIELD2: h1 OR
FIELD3: h1 OR FIELD4: h1 OR FIELD5: h1 OR FIELD6: h1 OR FIELD7: h1 ) OR ( (
FIELD1: b1 OR FIELD2: b1 OR FIELD3: b1 OR FIELD4: b1 OR FIELD5: b1 OR FIELD6: b1
OR FIELD7: b1 ) OR ( ( FIELD1: "h1 / b1" OR FIELD2: "h1 / b1" OR FIELD3: "h1 /
b1" OR FIELD4: "h1 / b1" OR FIELD5: "h1 / b1" OR FIELD6: "h1 / b1" OR FIELD7: "h1
/ b1" ) OR ( ( FIELD1: h OR FIELD2: h OR FIELD3: h OR FIELD4: h OR FIELD5: h OR
FIELD6: h OR FIELD7: h ) OR ( FIELD1: "b 1" OR FIELD2: "b 1" OR FIELD3: "b 1" OR
FIELD4: "b 1" OR FIELD5: "b 1" OR FIELD6: "b 1" OR FIELD7: "b 1" ) ) ) ) ) ) )
Time taken : 78696 ms
Query : ( ( ( FIELD1: relationship OR FIELD2: relationship OR FIELD3:
relationship OR FIELD4: relationship OR FIELD5: relationship OR FIELD6:
relationship OR FIELD7: relationship ) OR ( ( FIELD1: liability OR FIELD2:
liability OR FIELD3: liability OR FIELD4: liability OR FIELD5: liability OR
FIELD6: liability OR FIELD7: liability ) OR ( ( FIELD1: banks OR FIELD2: banks OR
FIELD3: banks OR FIELD4: banks OR FIELD5: banks OR FIELD6: banks OR FIELD7: banks
) OR ( ( FIELD1: mutual OR FIELD2: mutual OR FIELD3: mutual OR FIELD4: mutual OR
FIELD5: mutual OR FIELD6: mutual OR FIELD7: mutual ) OR ( ( FIELD1: funds OR
FIELD2: funds OR FIELD3: funds OR FIELD4: funds OR FIELD5: funds OR FIELD6: funds
OR FIELD7: funds ) OR ( ( FIELD1: sales OR FIELD2: sales OR FIELD3: sales OR
FIELD4: sales OR FIELD5: sales OR FIELD6: sales OR FIELD7: sales ) OR ( FIELD1:
hni OR FIELD2: hni OR FIELD3: hni OR FIELD4: hni OR FIELD5: hni OR FIELD6: hni OR
FIELD7: hni ) ) ) ) ) ) ) ) AND RANGE: ([ 0200 TO 0900 ]) AND REFIDa: (44 OR
44 OR 45 ) AND C_IDa: (14 )
Time taken : 66183 ms
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
