A question on performance

2015-01-07 Thread rama44ster
Hi, I have a lucene index which has close to 480M documents. And I ran around 1000 queries against the index. Each query is a boolean query with 3 different tokens. That is the query has 3 operands which MUST occur. Executing such 3 token queries gives the following latency percentiles. 50 = 16 ms

Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Clemens Wyss DEV
Say I wanted to find documents which have no content in "field1" (or dosuments that have no field 'field1'), wouldn't that be the following query? -field1:[* TO *] Thanks for you help Clemens - To unsubscribe, e-mail: java-user-

RE: A question on performance

2015-01-07 Thread Toke Eskildsen
rama44ster [rama44s...@gmail.com] wrote: [3 MUST clauses] > 50 = 16 ms > 75 = 52 ms > 90 = 121 ms > 95 = 262 ms > 99 = 76010 ms > 99.9 = 76037 ms > Is the latency expected to degrade when the number of docs is as high as > 480M? Try plotting response times as a function of hit count. My guess i

Re: A question on performance

2015-01-07 Thread Arvind Kalyan
Performance measurements must be made carefully. Have you performed any warmup? I recommend doing 10k calls just to let the dust settle including stuff like jit, before taking any kind if measurements. Also use mmapdirectory, if not already, to help with spikes in disk accesses. Also keep track

Re: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Erick Erickson
Should be, but it's a bit confusing because the query syntax is not pure boolean, so there's no set to take away the docs with entries in field 1, you need the match all docs bit, i.e. *:* -field1:[* TO *] (That's asterisk:asterisk -field1:[* TO *] in case the silly list interprets the asterisks a

AW: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Clemens Wyss DEV
Thx! -Ursprüngliche Nachricht- Von: Erick Erickson [mailto:erickerick...@gmail.com] Gesendet: Mittwoch, 7. Januar 2015 17:31 An: java-user Betreff: Re: Looking for docs that have certain fields empty (an/or not set) Should be, but it's a bit confusing because the query syntax is not pure

Re: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Ahmet Arslan
Hi Clemens, Since you are a lucene user, you might be interested in Uwe's response on a similar topic : http://find.searchhub.org/document/abb73b45a48cb89e Ahmet On Wednesday, January 7, 2015 6:30 PM, Erick Erickson wrote: Should be, but it's a bit confusing because the query syntax is not

Re: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Jack Krupansky
The pure negative query should work fine as a top level query - it's just when nested as a sub-query within parentheses that it misbehaves. -- Jack Krupansky On Wed, Jan 7, 2015 at 11:30 AM, Erick Erickson wrote: > Should be, but it's a bit confusing because the query syntax is not > pure boole

Re: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Jack Krupansky
Oops... I take that back! After I clicked Send I realized that this is the Lucene list - what I said is true for Solr queries, but that is because Solr added a "hack" to do things properly, but the Lucene query parser doesn't have that hack, so Erick is correct. -- Jack Krupansky On Wed, Jan 7, 2

RE: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Uwe Schindler
Hi, In Lucene you don't need to use a query parser for that, especially because range Queries is suboptimal and slow: There is already a very fast query/filter available. Ahmet Arslan already mentioned that, we had the same discussion a few weeks ago: http://find.searchhub.org/document/abb73b45

RE: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Chris Hostetter
: In Lucene you don't need to use a query parser for that, especially : because range Queries is suboptimal and slow: There is already a very : fast query/filter available. Ahmet Arslan already mentioned that, we had : the same discussion a few weeks ago: : http://find.searchhub.org/document/a

AW: Looking for docs that have certain fields empty (an/or not set)

2015-01-07 Thread Clemens Wyss DEV
Thanks for the input. In fact, we are using this very query as a Solr queryFilter (i.e. qf). Is there a specific (Lucene query) syntax to formulate (force the usage of) such a FieldValueFilter? -Ursprüngliche Nachricht- Von: Ahmet Arslan [mailto:iori...@yahoo.com.INVALID] Gesendet: Mitt