Re: Solr and Garbage Collection

t while your app is running affects your apps > performance - in that case you might want to look into turning on the > incremental mode. But you havn't mentioned that, so I doubt it. > > > > -- > - Mark > > http://www.lucidimagination.com > > > > Jonathan Arie

Re: Solr and Garbage Collection

tself. Upgrading to the latest release might help. Switching to a > different garbage collector should help. > > Bill > > On Sat, Sep 26, 2009 at 4:31 PM, Mark Miller > wrote: > > > Jonathan Ariel wrote: > > > Ok. After the server ran for more than 12 hours,

Re: Solr and Garbage Collection

Sun, Sep 27, 2009 at 5:05 PM, Mark Miller wrote: > Jonathan Ariel wrote: > > Well.. it is strange that when I use the default GC I don't get any > errors. > > > Not so strange - it's different code. The bug is Likely in the low pause > collector and not the serial

Re: Solr and Garbage Collection

Well.. it is strange that when I use the default GC I don't get any errors. If I'm so close to run out of memory I should see those OOM exceptions as well with the standard GC.BTW I'm faceting on around 13 fields and my total number of unique values is around 3. One of the fields with the bigge

Re: Solr and Garbage Collection

Yes, it seems like a bug. I will update my JVM, try again and let you know the results :) On 9/26/09, Mark Miller wrote: > Jonathan Ariel wrote: >> Ok. After the server ran for more than 12 hours, the time spent on GC >> decreased from 11% to 3,4%, but 5 hours later it crash

Re: Solr and Garbage Collection

pping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3 Memory: 4k page, physical 16432184k(92220k free), swap 2096472k(2065524k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (10.0-b22) for linux-amd64 JRE (1.6.0_06-b02), built on Mar 25 2008 01:03:02 by "java_re" with gcc 3.2.2 (Su

Re: Solr and Garbage Collection

many docs do you have ? I'm guessing its mostly FieldCache type > stuff, and thats the type of thing you can't really side step, unless > you give up the functionality thats using it. > > Grant Ingersoll wrote: > > > > On Sep 25, 2009, at 9:30 AM, Jonathan Ariel wrot

Re: Solr and Garbage Collection

compromise. First you mark whats reachable, then > everything thats marked is copied/compacted to the bottom of the heap. > Its all part of a "collection" though. > > Jonathan Ariel wrote: >> Maybe what's missing here is how did I get the 11%.I just ran solr with >&g

Re: FW: Solr and Garbage Collection

> -Fuad > http://www.linkedin.com/in/liferay > > > > -Original Message- > > From: Jonathan Ariel [mailto:ionat...@gmail.com] > > Sent: September-25-09 3:36 PM > > To: solr-user@lucene.apache.org > > Subject: Re: FW: Solr and Garbage Collection > &

Re: FW: Solr and Garbage Collection

I'm not planning on lowering the heap. I just want to lower the time "wasted" on GC, which is 11% right now.So what I'll try is changing the GC to -XX:+UseConcMarkSweepGC On Fri, Sep 25, 2009 at 4:17 PM, Fuad Efendi wrote: > Mark, > > what if piece of code needs 10 contiguous Kb to load a docume

Re: Solr and Garbage Collection

Solr, > > zero may be the right choice, since the HTTP cache is cherry-picking the > > easily cacheable requests. > > > > Note that a commit nearly doubles the memory required, because you have > two > > live Searcher objects with all their caches. Make sure you ha

Re: Solr and Garbage Collection

BTW why making them equal will lower the frequency of GC? On 9/25/09, Fuad Efendi wrote: >> Bigger heaps lead to bigger GC pauses in general. > > Opposite viewpoint: > 1sec GC happening once an hour is MUCH BETTER than 30ms GC once-per-second. > > To lower frequency of GC: -Xms4096m -Xmx4096m (ma

Re: Solr and Garbage Collection

I can't really understand how increasing the heap will decrease the 11% dedicated to GC On 9/25/09, Fuad Efendi wrote: >> You are saying that I should give more memory than 12GB? > > > Yes. Look at this: > >> > SEVERE: java.lang.OutOfMemoryError: Java heap space >> > org.apache.lucene.search.Fiel

Re: Solr and Garbage Collection

pecific heavily distributed > field values (cardinality is high! Size is 100bytes!) you need > 10,000,000,000 bytes for just this instance of FieldCache. > > GC does not play any role. FieldCache won't be GC-collected. > > > -Fuad > http://www.linkedin.com/in/liferay >

Re: Solr and Garbage Collection

ector$Connection.run(SocketConnector.java:226) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) On Fri, Sep 25, 2009 at 10:55 AM, Yonik Seeley wrote: > On Fri, Sep 25, 2009 at 9:30 AM, Jonathan Ariel > wrote: > > Hi to all! > > Lately my so

Solr and Garbage Collection

Hi to all! Lately my solr servers seem to stop responding once in a while. I'm using solr 1.3. Of course I'm having more traffic on the servers. So I logged the Garbage Collection activity to check if it's because of that. It seems like 11% of the time the application runs, it is stopped because of

Very big numbers

Hi! I need to index in solr very big numbers. Something like 99,999,999,999,999.99 Right now i'm using an sdouble field type because I need to make range queries on this field. The problem is that the field value is being returned in scientific notation. Is there any way to avoid that? Thanks! Jon

Re: Single Core or Multiple Core?

Yes, I think it is better to be backward compatible or the impact of moving to the new solr version would be big. On Mon, Sep 14, 2009 at 12:24 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > On Mon, Sep 14, 2009 at 8:16 PM, Uri Boness wrote: > > > Is it really a problem? I mean, a

Re: Single Core or Multiple Core?

What do you mean by "single-core deployments does not have a way to enable CoreAdminHandler"?I'm just trying to understand the feature that you are talking about On Sat, Sep 12, 2009 at 6:44 AM, Uri Boness wrote: > +1 > Can you add a JIRA issue for that so we can vote for it? > > > Chris Hostett

Re: Very slow first query

ot the number of documents, but the > total size in gigabytes in solr/data/index. > > On Fri, Sep 11, 2009 at 10:21 AM, Jonathan Ariel > wrote: > > Ok thanks, if it's the IO OS Disk cache, which would be my options? > changing > > the disk to a faster one? &g

Re: Very slow first query

zation during this 50s query. > > -Yonik > http://www.lucidimagination.com > > > > On Fri, Sep 11, 2009 at 8:59 AM, Jonathan Ariel > wrote: > > yes of course. but in my case I'm not using filter queries nor facets. > > it is a really simple query. actually the query para

Re: Very slow first query

i Boness wrote: > "Not having any facet" and "Not using a filter cache" are two different > things. If you're not using query filters, you can still have facet > calculated and returned as part of the search result. The facet component > uses lucene's f

Re: Query runs faster without filter queries?

amount of time, right? On Thu, Sep 10, 2009 at 9:44 PM, Jonathan Ariel wrote: > Thanks! I don't think I can use an unreleased version of solr even is it's > stable enough (crazy infrastructure guys) but I might be able to apply the 2 > patches mentioned in the link you sent.

Re: Very slow first query

eries by configuring a static warming query in > solrconfig.xml that includes the common sorts and facets. > > -Yonik > http://www.lucidimagination.com > > On Thu, Sep 10, 2009 at 8:55 PM, Jonathan Ariel > wrote: > > Hi!Why would it take for the first query that I execute

Very slow first query

Hi!Why would it take for the first query that I execute almost 60 seconds to run and after that no more than 50ms? I disabled all my caching to check if it is the reason for the subsequent fast responses, but the same happens. I'm using solr 1.3. Something really strange is that it doesn't happen w

Re: Query runs faster without filter queries?

ter would be applied. AFAIK this is fixed in 1.4 where now the > documents which are defined by the filters are skipped during the query > execution. > > Uri > > > Jonathan Ariel wrote: > >> Hi all! >> I'm trying to measure the query response time when using

Re: Single Core or Multiple Core?

angar < shalinman...@gmail.com> wrote: > On Fri, Sep 4, 2009 at 4:35 AM, Jonathan Ariel wrote: > > > It seems like it is really hard to decide when the Multiple Core solution > > is > > more appropriate.As I could understand from this list and wiki the > Multiple &g

Query runs faster without filter queries?

Hi all! I'm trying to measure the query response time when using just a query and when using some filter queries. From what I read and understand adding filter query should boost the query response time. I used luke to understand over which fields I should use filter query (those that have few uniq

Single Core or Multiple Core?

It seems like it is really hard to decide when the Multiple Core solution is more appropriate.As I could understand from this list and wiki the Multiple Core feature was designed to address the need of handling different sets of data within the same solr instance, where the sets of data don't need

Possible bug with sdouble?

Hi everyone! So it seems like I ran into a bug with sdouble. I have a document with two fields. field1 is double when I index I send for both of them the same value (7300) and when I'm querying I'm getting the following 7300 7.3E7 What do you think? Why am I getting different notati

Stemmer vs. exact match

Hi! I'm wondering what solr is really doing with the exact word vs. the stemmed word. So for example I have 2 documents. The first one has in the title the word "convertible" The second one has "convert" When solr stem the titles, both will be the same since convertible -> convert. Then when I sea

Re: Getting a document by primary key

er.getFirstMatch(t); On Mon, Oct 20, 2008 at 1:16 PM, Otis Gospodnetic < [EMAIL PROTECTED]> wrote: > Jonathan, http://./.?q=: should > be fine. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message

Getting a document by primary key

I'm developing my own request handler and given a document primary key I would like to get it from the index. Which is the best and fastest way to do this? I will execute this request handler several times and this should work really fast. Sorry if it's a basic question. Thanks! Jonathan

Expansion stemming

hi, I was looking into the solr wiki and saw this: "Expansion stemming — Takes a root word and 'expands' it to all of its various forms — can be used either at insertion time or at query time." How do I specify that I want the expansion stemming instead of the porter stemming? Thanks!

Re: Bulk delete

able in a distributed system to perform requests of > unbounded size (not to say that it won't work). If the concern is > throughput, large batches should be sufficient. > > -Mike > > > On 4-Jul-08, at 9:06 AM, Jonathan Ariel wrote: > > Yes, I just wanted to avoid N re

Re: Bulk delete

oh. you're right! if using 1.13 and if there is no limit to the amount of ids I can send with the delete tag. On Fri, Jul 4, 2008 at 1:10 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote: > On Fri, Jul 4, 2008 at 12:06 PM, Jonathan Ariel <[EMAIL PROTECTED]> > wrote: > >

Re: Bulk delete

Yes, I just wanted to avoid N requests and do just 2. On Fri, Jul 4, 2008 at 12:48 PM, Walter Underwood <[EMAIL PROTECTED]> wrote: > Send multiple deletes, with a commit after the last one. --wunder > > On 7/4/08 8:40 AM, "Jonathan Ariel" <[EMAIL PROTECTED]> wrote:

Re: Bulk delete

this is a really nice feature. any known limit for the amount of Ids that I can add there? On Fri, Jul 4, 2008 at 12:40 PM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > yeah I know. the problem with a query is that there is a maximum amount of > query terms that I can add, which i

Re: Bulk delete

ery or by an id. It is like you use any > database . If you can find a condition by which you can identify these > docs then you can delete by a query . > --Noble > On Fri, Jul 4, 2008 at 8:22 PM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > > Hi, > > Is there any

Bulk delete

Hi, Is there any good way to do a bulk delete of several documents? I have more than 1000 documents to delete... and I don't want to send N request with X. Doing a query delete isn't a good solution because I have a maximum amount of terms that I can use in the query. For example: id:(X1 OR X2 OR .

Re: NGramTokenizer issue

icatesTokenFilterFactory. Removing duplicates might change the structure of the word. On Thu, Jun 26, 2008 at 12:25 AM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > Well, it is working if I search just two letters, but that just tells me > that something is wrong somewhere. > The Anal

Re: NGramTokenizer issue

an environment issue. Maybe I am doing something wrong. Any thoughts on that? Thanks! Jonathan On Wed, Jun 25, 2008 at 9:44 PM, Norberto Meijome <[EMAIL PROTECTED]> wrote: > On Wed, 25 Jun 2008 15:37:09 -0300 > "Jonathan Ariel" <[EMAIL PROTECTED]> wrote: > > >

NGramTokenizer issue

Hi, I've been trying to use the NGramTokenizer and I ran into a problem. It seems like solr is trying to match documents with all the tokens that the analyzer returns from the query term. So if I index a document with a title field with the value "nice dog" and search for "dog" (where the NGramtoke

Re: Getting maximum and minimum values of a field

e my own foo function >> and plug it in somewhere? I'm theorizing though... I'll look at the patch >> later. >> >> Otis >> -- >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> >> - Original Message >>

Re: Getting maximum and minimum values of a field

anged at all. It reads just once the value of the field for each > document selected, so no extra loops either in or outside the main loop ;). > > Pako > > > Jonathan Ariel wrote: > >> Well, this is the first version of the patch. >> https://issues.apache.org/jira/bro

Re: Getting maximum and minimum values of a field

Well, this is the first version of the patch. https://issues.apache.org/jira/browse/SOLR-588 Give me your feedback so we can make it better. Thanks! On Fri, May 30, 2008 at 10:36 AM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > What do you mean? Right now you send an extra

Re: Getting maximum and minimum values of a field

tting maximum and minimum values of a field > > > > I know that I would definately be interested in that. :) > > > > /Jimi > > > > Quoting Jonathan Ariel : > > > > > Ok. So I have a version of solr with a small modification to the > SimpleFacet >

Re: Getting maximum and minimum values of a field

ate a patch and add it to JIRA. What do you think? Jonathan On Wed, May 28, 2008 at 12:58 AM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > Hi!I'm looking for a way to get the maximum and minimum value of a field > (like price) in a specific query. First I thought about using facet

Getting maximum and minimum values of a field

Hi!I'm looking for a way to get the maximum and minimum value of a field (like price) in a specific query. First I thought about using facet for that, but since price may have lots of different and unique values, a facet is not a good idea. Any thoughts about how can I achieve this? Thanks! Jonat

Re: How to limit number of pages per domain

t; Subject: Re: How to limit number of pages per domain > > > > There is a documentation: > > > > http://wiki.apache.org/solr/FieldCollapsing > > > > Koji > > > > Jonathan Ariel wrote: > > > Sorry. But how field collapsing works? Is there documentation about > this > > > anywhere? Thanks! > > > > >

Re: How to limit number of pages per domain

Sorry. But how field collapsing works? Is there documentation about this anywhere? Thanks! On Wed, May 21, 2008 at 7:02 PM, Chris Hostetter <[EMAIL PROTECTED]> wrote: > : > : I'm indexing pages from multiple domains. In any given > : result set, I don't want to return more than two links > : from

Re: Reindexing mode for solr

on the slave. If > your concern is that the index will be sent to the search slave while you > are still reindexing, just don't commit until you are done. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > - Original Message ---

Reindexing mode for solr

Hi, Is there any way to tell solr to load in a kind of reindexing mode, which won't open a new searcher after every commit, etc? This is just when you don't have it available to query because you just want to reindex all the information. What do you think? Jonathan

Re: More Like This boost

Ok. Here it is. https://issues.apache.org/jira/browse/LUCENE-1272 On Tue, Apr 22, 2008 at 2:24 PM, Francisco Sanmartin <[EMAIL PROTECTED]> wrote: > Yep, it would be nice for MLT to have this feature, that's why I am trying > to do it from the querys before sending the query to Solr. These are

Re: MoreLikeThis patch to support boost factor

Ok. Posted. You'll find a patch with unit test. https://issues.apache.org/jira/browse/LUCENE-1272 Thanks! On Wed, Apr 23, 2008 at 10:25 PM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > Yes. Sure. I'll do that. Just wanted some feedback before posting it. As > soon as I do

Re: MoreLikeThis patch to support boost factor

o have a unit test one > could run to see how/that this works? > > Thanks, > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > From: Jonathan Ariel <[EMAIL PROTECTED]> > To: solr-user@lucene.a

MoreLikeThis patch to support boost factor

This is a patch I made to be able to boost the terms with a specific factor beside the relevancy returned by MoreLikeThis. This is helpful when having more then 1 MoreLikeThis in the query, so words in the field A (i.e. Title) can be boosted more than words in the field B (i.e. Description). Any f

Re: config for very frequent solr updates

Ok. So it is there, in the admin page you can see: openedAt : Fri Apr 18 10:30:02 EDT 2008 registeredAt : Fri Apr 18 10:30:31 EDT 2008 That means that the warmup took around 29 seconds. Thanks! On Fri, Apr 18, 2008 at 11:17 AM, Geoffrey Young <[EMAIL PROTECTED]> wrote: > > > Otis Gospodnetic wr

Re: config for very frequent solr updates

Is there any way to measure how long it takes to warmup? On Fri, Apr 18, 2008 at 8:29 AM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > And how would you combine the results from the different indexes? > > > On Fri, Apr 18, 2008 at 12:40 AM, Otis Gospodnetic < >

Re: config for very frequent solr updates

And how would you combine the results from the different indexes? On Fri, Apr 18, 2008 at 12:40 AM, Otis Gospodnetic < [EMAIL PROTECTED]> wrote: > Geoff, > > There was just another thread where the person said he was doing updates > every 2 minutes. Like you said, with the way Solr warms searche

Re: too many queries?

> > >> I would start with 8GB of RAM. 4GB for your index, 2 for Solr, 1 for > >> the OS and 1 for other processes. That might be tight. 12GB would > >> be a lot better. > >> > >> wunder > >> > >> On 4/16/08 7:50 AM, "Jonatha

Re: too many queries?

Is there anyway to know how much memory is being used in caches? On Wed, Apr 16, 2008 at 11:50 AM, Jonathan Ariel <[EMAIL PROTECTED]> wrote: > In order to do that I have to change to a 64 bits OS so I can have more > than 4 GB of RAM.Is there any way to see how long does it take

Re: too many queries?

the Solr caches are flushed > before they even start to stabilize. Two things to try: > > * commit less often, 5 minutes or 10 minutes > * have enough RAM that your entire index can fit in OS file buffers > > wunder > > On 4/16/08 6:27 AM, "Jonathan Ariel" <[E

Re: too many queries?

ition the data to a static set and a dynamic set, and then combining them at query time? Do you have a link to read about that? On Tue, Apr 15, 2008 at 7:21 PM, Mike Klaas <[EMAIL PROTECTED]> wrote: > On 15-Apr-08, at 5:38 AM, Jonathan Ariel wrote: > > > My index is 4GB on disk.

Re: too many queries?

; >Erik > > > > > On Apr 15, 2008, at 8:38 AM, Jonathan Ariel wrote: > > > filterCache > > autowarmCount=256 > > lookups : 24241 > > hits : 21575 > > hitratio : 0.89 > > inserts : 3708 > > evictions : 3155 > > size : 512 > >

Re: too many queries?

izes like and how is the warm-up configured? > Is there any IO on the slaves? (run vmstat or iostat or some such) > How is the CPU usage looking? > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > - Original Message > From: Jonathan Ar

too many queries?

Hi, I have some questions about performance for you guys. So basically I have 2 slave solr servers and 1 master solr server load balanced and around 100 request/second, aprox. 50 request per second per solr server. My index is about 4 million documents and the average query response time is 0.6 sec

Re: Date range performance

Ok. Just to give some feedback. I reindexed with less precision as you told me and it's working really fast. Thanks for your help! Jonathan On Fri, Apr 4, 2008 at 6:02 PM, Chris Hostetter <[EMAIL PROTECTED]> wrote: > > : Looking into the code it seems like a Lucene problem, more than Solr. It >

Re: Date range performance

n 3-Apr-08, at 4:24 PM, Jonathan Ariel wrote: > > > Is this depends on the number of documents that matches the query or the > > number of documents in the index? > > > > This aspect is more depedent on the number of terms that the date query > translates into. >

Re: Date range performance

wrote: > > On 3-Apr-08, at 2:14 PM, Jonathan Ariel wrote: > > > Hi, > > I'm experiencing a really poor performance when using date ranges in > > solr > > query. Is it a know issue? is there any special consideration when using > > date ranges? It s

Date range performance

Hi, I'm experiencing a really poor performance when using date ranges in solr query. Is it a know issue? is there any special consideration when using date ranges? It seems weird because I always thought date dates are translated to strings, so internally lucene resolves everything the same way. So

Embedded Solr and Servlet Solr

Hi! I am using Solr (the classic servlet one) in my application. But for special cases, and just for the sake of optimization, I would like to access it locally so I can avoid HTTP calls and XML serializing. This is for full re indexation of Solr, which is much faster in embedded environment than s

Re: multivalued dynamic fields performance

Thanks! So there is just one penalty when faceting, which is my case. "TermEnum is good for a limited number of different indexed terms in the field, and allows multiple terms per field per document" How much is limited number of different indexed terms in the field? If I'm going to have nearly alw

multivalued dynamic fields performance

Hi, Do you know if there is a performance impact when using multivalued dynamic fields when it's not always necessary to store more than one value? Since I'm going to add dynamic fields to my schema and I'm not sure if the field will be multivalued or not, I thought about doing them multivalued. In

Re: Successful project based on SOLR

ras that Solr brings to a Lucene index. > > - Charlie > > -Original Message- > From: Jonathan Ariel [mailto:[EMAIL PROTECTED] > Sent: Thursday, December 20, 2007 12:49 PM > To: solr-user@lucene.apache.org > Subject: Re: Successful project based on SOLR > > What's

Re: Successful project based on SOLR

What's the difference with that and Hibernate Search ? On Dec 20, 2007 2:09 PM, Charlie Jackson <[EMAIL PROTECTED]> wrote: > Congratulations! > > > It uses an custom hibernate-SOLR > bridge which allows transparent persistence of entities on different > SOLR ser

Solr PHP client

Hi! I'm wondering if someone is using a PHP client for solr. Actually I'm not sure if there is one out there. Would you be interested in having a SolrJ port for PHP? Thanks, Jonathan Leibiusky

Random return

Hi! I need to query solr and then return a random document from the result set. Any ideas on how can I achieve this? Thanks! Jonathan

Re: Solr and terracotta

I'm looking forward for this implementation! I think it'll be really great feature! Does anybody knows how long it takes terracotta sync a RAMDir? Can it be configured? What's the chance of commiting one document on server A and querying on server B that wasn't synchronized yet? On 8/27/07, Je

Re: Solr and terracotta

If I am not wrong once you have the RAMDir feature mounting Terracotta should be transparent and fast, right? On 8/22/07, Orion Letizi <[EMAIL PROTECTED]> wrote: > > > Jeryl, > > I remember you asking about how to hook in the RAMDirectory a while back. > It seemed like there was maybe some support

Re: Solr and terracotta

How come it didn't work? How did you add RAMDir support to solr? On 8/22/07, Jeryl Cook <[EMAIL PROTECTED]> wrote: > > tried it, didn't work that well...so I ended up making my own little > faceted Search engine directly using RAMDirectory and clustering it via > Terracotta...not as good as SOLR(s

Solr and terracotta

Recently I ran into this topic. I googled it a little and didn't find much information. It would be great to have solr working with RAMDirectory and Terracotta. We could stop using crons for rsync, right? Has anyone tried that out?