Re: Slow queries for common terms

2013-03-25 Thread Erick Erickson
-West [mailto:tburt...@umich.edu] Sent: Saturday, March 23, 2013 1:38 AM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries because our positions lists are so

RE: Slow queries for common terms

2013-03-25 Thread David Parks
] Sent: Monday, March 25, 2013 10:04 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms take a look here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html looking at memory consumption can be a bit tricky to interpret with MMapDirectory. But you say I

RE: Slow queries for common terms

2013-03-23 Thread David Parks
-Original Message- From: Tom Burton-West [mailto:tburt...@umich.edu] Sent: Saturday, March 23, 2013 1:38 AM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries

Re: Slow queries for common terms

2013-03-22 Thread Jan Høydahl
users. If you could elaborate a bit on your thinking I'd be quite grateful. David -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 8:01 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi

Re: Slow queries for common terms

2013-03-22 Thread Tom Burton-West
Hi David and Jan, I wrote the blog post, and David, you are right, the problem we had was with phrase queries because our positions lists are so huge. Boolean queries don't need to read the positions lists. I think you need to determine whether you are CPU bound or I/O bound.It is possible

Re: Slow queries for common terms

2013-03-21 Thread Jan Høydahl
Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2 and try out the approach using a dictionary of the most common words in your index. You don't say how many documents, avg. doc size, the IDF value of book,

RE: Slow queries for common terms

2013-03-21 Thread David Parks
. -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow

Re: Slow queries for common terms

2013-03-21 Thread Jan Høydahl
] Sent: Thursday, March 21, 2013 5:43 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, I think you can start by reading this blog http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-w ords-part-2 and try out the approach using

Re: Slow queries for common terms

2013-03-21 Thread Ahmet Arslan
Hi David, I happen to know CommonTermsQuery added recently. But now sure how to use it. http://lucene.apache.org/core/4_1_0/queries/org/apache/lucene/queries/CommonTermsQuery.html --- On Thu, 3/21/13, David Parks david.pa...@frugg.com wrote: From: David Parks david.pa...@frugg.com Subject:

RE: Slow queries for common terms

2013-03-21 Thread David Parks
for a few users. If you could elaborate a bit on your thinking I'd be quite grateful. David -Original Message- From: Jan Høydahl [mailto:jan@cominvent.com] Sent: Thursday, March 21, 2013 8:01 PM To: solr-user@lucene.apache.org Subject: Re: Slow queries for common terms Hi, If you say