Re: solr blocking on commit

2010-01-20 Thread Steve Conover
> How solr organized so that search can continue when a commit has closed the > index? > Also, looking at lucene docs, commit causes a system fsync().  Won't search > also > get blocked by the IO traffic generated? ...I'll run iostat too and see if there's anything interesting to report

Re: solr blocking on commit

2010-01-19 Thread Steve Conover
added in a long time / ever. Once a set of documents is posted to /update, if I re-post it solr behaves pretty well - and that's true even if I restart solr. On Tue, Jan 19, 2010 at 3:05 PM, Yonik Seeley wrote: > On Tue, Jan 19, 2010 at 5:57 PM, Steve Conover wrote: >> I'm using lat

solr blocking on commit

2010-01-19 Thread Steve Conover
I'm using latest solr 1.4 with java 1.6 on linux. I have a 3M document index that's 10+GB. We currently give solr 12GB of ram to play in and our machine has 32GB total. We're seeing a problem where solr blocks during commit - it won't server /select requests - in some cases for more than 15-30 s

Re: solr+jetty logging to syslog?

2009-11-21 Thread Steve Conover
Does no one send solr logging to syslog? On Thu, Nov 19, 2009 at 5:54 PM, Steve Conover wrote: > The solution involves slf4j to log4j to syslog (at least, for solr), > but I'm having some trouble stringing all the parts together.  If > anyone is doing this, would you mind posti

solr+jetty logging to syslog?

2009-11-19 Thread Steve Conover
The solution involves slf4j to log4j to syslog (at least, for solr), but I'm having some trouble stringing all the parts together. If anyone is doing this, would you mind posting how you use slf4j-log4j jar, what your log4j.properties looks like, what your java system properties settings are, and

Re: is it possible to speed up this query?

2009-10-04 Thread Steve Conover
> Lowering the precisionStep of the trie field could help.  With a > precisionStep of 8, there would be 256 top-level terms to step over. > If you lowered it to 6, it would cut it to 64 or 4 would cut it to 16. > > The fastest would be to index a separate field to indicate presence or > absence of

Re: is it possible to speed up this query?

2009-10-04 Thread Steve Conover
done? Anything I could be specifically caching? Regards, Steve On Sat, Oct 3, 2009 at 9:07 AM, Yonik Seeley wrote: > On Sat, Oct 3, 2009 at 11:44 AM, Steve Conover wrote: >> Hi, >> >> I have a query that's taking about 120ms to run that I'd like to try >> t

is it possible to speed up this query?

2009-10-03 Thread Steve Conover
Hi, I have a query that's taking about 120ms to run that I'd like to try to reduce if possible. The key element that seems to be at the root of about 100ms of the delay is a "not null" criterion (it happens to be a Long field), i.e. foo:[* TO *] I've tried converting to trie ranges and that has

Re: trie fields and sortMissingLast

2009-10-01 Thread Steve Conover
> Not in time for 1.4, but yes they will eventually get it. > It has to do with the representation... currently we can't tell > between a 0 and "missing". Hmm. So does that mean that a query for latitudes, stored as trie floats, from -10 to +10 matches documents with no (i.e. null) latitude value

Re: trie fields and sortMissingLast

2009-10-01 Thread Steve Conover
I just noticed this comment in the default schema: Does that mean TrieFields are never going to get sortMissingLast? Do you all think that a reasonable strategy is to use a copyField and use "s" fields for sorting (only), and trie for everything else? On Wed, Sep 30, 2009 at 10:59

trie fields and sortMissingLast

2009-09-30 Thread Steve Conover
Am I correct in thinking that trie fields don't support sortMissingLast (my tests show that they don't). If not, is there any plan for adding it in? Regards, Steve

Re: recommendation for document store to use alongside Solr?

2009-05-26 Thread Steve Conover
It's not distributed, but I like Tokyo Cabinet. Tokyo Tyrant adds http on top, I've never used Tyrant though. -Steve On Tue, May 26, 2009 at 7:50 AM, Peter Keane wrote: > Hi All- > > I've just recently began playing with Apache Solr, and it seems to be > a perfect fit for our project (http://co

Re: optimization advice?

2009-03-28 Thread Steve Conover
aracters?   > If so, you could create a separate field for that (copyField will come handy) > and that should consume a little less memory. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message >>

Re: optimization advice?

2009-03-27 Thread Steve Conover
be indexed.   > I have a hunch your name field is tokenized. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message >> From: Steve Conover >> To: solr-user@lucene.apache.org >> Sent: Friday, Marc

Re: optimization advice?

2009-03-27 Thread Steve Conover
ery granular, and if they are, do you really need them to be that precise? > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message >> From: Steve Conover >> To: solr-user@lucene.apache.org >> Sent: Fr

Re: optimization advice?

2009-03-27 Thread Steve Conover
> Steve, > > Maybe you can tell us about: sure > - your hardware 2.5GB RAM, pretty modern virtual servers > - query rate Let's say a few queries per second max... < 4 And in general the challenge is to get latency on any given query down to something very low - we don't have to worry about a

optimization advice?

2009-03-26 Thread Steve Conover
Hi, I've looked over the public Solr perf docs and done some searching on this mailing list. Still, I'd like to seek some advice based on my specific situation: - 2-3 million documents / 5GB index - each document has 40+ indexed fields, and many multivalue fields - only primary keys are "stored"

Re: commit / new searcher delay?

2009-03-05 Thread Steve Conover
That's exactly what I'm doing, but I'm explicitly replicating, and committing. Even under these circumstances, what could explain the delay after commit before the new index becomes available? On Thu, Mar 5, 2009 at 10:55 AM, Shalin Shekhar Mangar wrote: > On Thu, Mar 5, 2009

Re: commit / new searcher delay?

2009-03-05 Thread Steve Conover
Yep, I notice the default is true/true, but I explicitly specified both those things too and there's no difference in behavior. On Wed, Mar 4, 2009 at 7:39 PM, Shalin Shekhar Mangar wrote: > On Thu, Mar 5, 2009 at 6:06 AM, Steve Conover wrote: > >> I'm doing some testing

commit / new searcher delay?

2009-03-04 Thread Steve Conover
I'm doing some testing of a solr master/slave config and find that, after syncing my slave, I need to sleep for about 400ms after commit to "see" the new index state. i.e. if I don't sleep, and I execute a query, I get results that reflect the prior state of the index. I suspect this has somethin

advice on minimal solr/jetty

2009-01-19 Thread Steve Conover
Hi everyone, I'd like to see how much I can reduce the startup time of jetty/solr. Right now I have it at about 3s - that's fast, but I'd like to see how close to zero I can get it. I've minimized my schema and solrconfig down to what I use (my solr needs are pretty vanilla). Now I'm looking at