Re: Solr and jvm Garbage Collection tuning

Hi Tim, For what it is worth, behind Trove (http://trove.nla.gov.au/) are 3 SOLR-managed indices and 1 Lucene index. None of ours is as big as one of your shards, and one of our SOLR-managed indices is tiny, but your experiences with long GC pauses are familar to us. One of the most difficult in

Re: How to extend IndexSchema and SchemaField

How about this: value It generally would be better to keep the attribute space clean and make it very clear you are doing something unique to this field. On Fri, Sep 10, 2010 at 9:16 AM, Renaud Delbru wrote: >  Hi Charlie, > > On 10/09/10 16:11, Charlie Jackson wrote: >> >> Have you alrea

multivalued fields in result

Is it possible to return multivalued files in the result? I would like to have a multivalued field that is stored and not indexed (I also copy the same field into another field where it is tokenized and indexed). I would then like all the values of this field returned in the result set. Is

Re: Null Pointer Exception with shards&facets where some shards have no values for some facets.

On Fri, Sep 10, 2010 at 7:21 PM, Ron Mayer wrote: > Ron Mayer wrote: > Yes, looks good now. > Thanks! Great, thanks for the report! -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8

Re: Null Pointer Exception with shards&facets where some shards have no values for some facets.

Ron Mayer wrote: > Yonik Seeley wrote: >> I just checked in the last part of those changes that should eliminate >> any restriction on key. >> But, that last part dealt with escaping keys that contained whitespace or } >> Your example really should have worked after my previous 2 commits. >> Perhap

Solr and jvm Garbage Collection tuning

We have noticed that when the first query hits Solr after starting it up, memory use increases significantly, from about 1GB to about 16GB, and then as queries are received it goes up to about 19GB at which point there is a Full Garbage Collection which takes about 30 seconds and then memory use

Solr memory use, jmap and TermInfos/tii

Hi all, When we run the first query after starting up Solr, memory use goes up from about 1GB to 15GB and never goes below that level. In debugging a recent OOM problem I ran jmap with the output appended below. Not surprisingly, given the size of our indexes, it looks like the TermInfo and T

Re: Inconsistent search results with multiple keywords

Stéphane Corlosquet wrote: > Hi all, > > I'm new to solr so please let me know if there is a more appropriate place > for my question below. > > I'm noticing a rather unexpected number of results when I add more keywords > to a search. I'm listing below a example (where I replaced the real keywor

RE: How to Update Value of One Field of a Document in Index?

The MoreLikeThis component actually can accept external input: http://wiki.apache.org/solr/MoreLikeThisHandler#Using_ContentStreams   -Original message- From: Jonathan Rochkind Sent: Fri 10-09-2010 18:59 To: solr-user@lucene.apache.org; Subject: RE: How to Update Value of One Field of a

Re: Autocomplete with Filter Query

Hi there, I don't know if my idea is perfect but it seems to work ok in my twitter-search prototype: http://www.jetwick.com (keep in mind it is a vhost and only one fat index, no sharding, etc... so performance isn't perfect ;-)) That said, type in 'so' and you will get 'soldier', 'solar', ... bu

Re: Solr CoreAdmin create ignores dataDir Parameter

Frank, have a look at SOLR-646. Do you think a workaround for the data-dir-tag in the solrconfig.xml can help? I think about something like ${solr./data/corename} for illustration. Unfortunately I am not very skilled in working with solr's variables and therefore I do not know what variables ar

Re: SEVERE: java.io.IOException: The specified network name is no longer available

Erick Erickson wrote: > > SOLR uses Lucene, so I'd expect that putting your index files on a share > isn't any more robust under SOLR than Lucene > > Sounds to me like your network's glitchy. > > Except that with Lucene we had separate processes searching and indexing directly against th

Re: SEVERE: java.io.IOException: The specified network name is no longer available

SOLR uses Lucene, so I'd expect that putting your index files on a share isn't any more robust under SOLR than Lucene Sounds to me like your network's glitchy. FWIW Erick On Fri, Sep 10, 2010 at 3:00 PM, brian519 wrote: > > > Yonik Seeley-2-2 wrote: > > > > > > Hmmm, that's interesting. I

Re: SEVERE: java.io.IOException: The specified network name is no longer available

Yonik Seeley-2-2 wrote: > > > Hmmm, that's interesting. I wonder if it's a Java bug or something? > There's nothing in lucene/solr that I know of that would lead to "The > specified network name is no longer > available". > > What JVM are you using? > > -Yonik > http://lucenerevolution.org

Re: SEVERE: java.io.IOException: The specified network name is no longer available

On Fri, Sep 10, 2010 at 2:12 PM, brian519 wrote: > Once we see the error, it is persistent.  Restarting Tomcat makes the error > stop.  This is happening across a variety of deployments and networks, so I > don't think there is an actual network problem.  Many other apps operate > fine on the same

SEVERE: java.io.IOException: The specified network name is no longer available

Hi all, Using Solr 1.4 hosted with Tomcat 6 on Windows 2003 (sometimes Windows 2008) Occasionally, we can't search anymore and this error shows up in the log file: SEVERE: java.io.IOException: The specified network name is no longer available at java.io.RandomAccessFile.readBytes(Native

Re: How to Update Value of One Field of a Document in Index?

I want to do MoreLikeThis to find documents that are similar to the document that I am indexing.  Then I want to calculate the average of one of the fields of all those documents and input this average into a field of the document that I am indexing.  From my research, it seems that MoreLikeThis

Sorting not working on a string field

Hello, I seem to be having a problem with sorting. I have a string field (time_code) that I want to order by. When the results come up, it displays the results differently from relevance which I would assume, but the results aren't ordered. The data in time_code came from a numeric decimal with

RE: Autocomplete with Filter Query

Cool idea! I was suggesting a copy field because I want to provide autocomplete on any field that the dismax can search on - eg if dismax searches both name and phone, then when they start typing name or phone, I want it to give autocompletion there So to get your idea clear are you suggesting a

RE: How to Update Value of One Field of a Document in Index?

"More like this" is intended to be run at query time. For what reasons are you thinking you want to (re-)index each document based on the results of MoreLikeThis? You're right that that's not what the component is intended for. Jonathan From: Savannah B

Re: Delta Import with something other than Date

> Can you provide a sample of passing the parameter via URL? And how using it > would look in the data-config.xml http://wiki.apache.org/solr/DataImportHandler#Accessing_request_parameters

Re: How to extend IndexSchema and SchemaField

Hi Charlie, On 10/09/10 16:11, Charlie Jackson wrote: Have you already explored the idea of using a custom analyzer for your field? Depending on your use case, that might work for you. Yes, I have thought of that, or even extending field type. But this does not work for my use case, since I ca

Re: How to extend IndexSchema and SchemaField

Hi Javier, On 10/09/10 07:15, Javier Diaz wrote: Looking at the code we found out that there's no way to extend the schema. Finally we copied part of the code that reads the schema in our RequestHandler. It works but I'm not sure if it's the best way to do it. Let me know if you want our code a

what would cause large numbers of executeWithRetry INFO messages?

I see a large number (~1000) of the following executeWithRetry messages in my apache catalina log files every day (see bolded snippet below). They seem to appear at random intervals. Since they are not flagged as errors or warnings, I have been ignoring them for now. However, I started wonderin

Re: Building query based on value of boolean field

Oh and the field in the result looks like: false but when I do this: q=partylocation:false I still get no results! :s -- View this message in context: http://lucene.472066.n3.nabble.com/Building-query-based-on-value-of-boolean-field-tp1449018p1453266.html Sent from the Solr - User mailing list

Re: No more trunk support for 2.9 indexes

If a tool exists for converting 2.9->3.0.x, it would likely be faster. Do you know if such a tool exists? Remaking the index, in my case, can only be done from the existing index because the original data is no longer available (it is transient network data). I suppose an index 'remaker' might be s

RE: Autocomplete with Filter Query

I've been thinking about this too, and haven't come up with any GREAT way. But there are several possible ways, that will do different things, good or bad, depending on the nature of your data and exactly what you want to do. So here are some ideas I've been thinking about, but not a ready made

Re: No more trunk support for 2.9 indexes

So converting an index would be faster than remaking it? Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Fri, 9/10/10, peter.stu...@gmail.com wrote: > From: peter.st

No more trunk support for 2.9 indexes

Hi, I'm sure there are good reasons for the decision to no longer support 2.9 format indexes in 4.0, and not have an automatic upgrade as in previous versions. Since Lucene 3.0.2 is 'out there', does this mean the format is nailed down, and some sort of porting is possible? Does anyone kn

Re: How to Update Value of One Field of a Document in Index?

Thanks.  I am trying to use MoreLikeThis in Solr to find similar documents in the solr index and use the data from these similar documents to modify a field in each document that I am indexing.  I found that MoreLikeThis in Solr only works when the document is in the index, is it true?  If so, I

Autocomplete with Filter Query

Hi, Is there any way to provide autocomplete while filtering results? Suppose I had a bunch of people and each person has multiple occupations. When I select 'Assistant' in a filter box, it would be nice if autocomplete only provides assistant names, instead of all names. The other issue is tha

RE: How to extend IndexSchema and SchemaField

Have you already explored the idea of using a custom analyzer for your field? Depending on your use case, that might work for you. - Charlie

Re: solr / lucene engineering positions in Boston, MA USA @ the Echo Nest

On Fri, Sep 10, 2010 at 9:18 AM, Brian Whitman wrote: > Hi all, brief message to let you know that we're in heavy hire mode at the > Echo Nest. As many of you know we are very heavy solr/lucene users (~1bn > documents across many many servers) and a lot of our staff have been working > with and co

Re: How to delete documents from a SOLR cloud / balance the shards in the cloud?

Stephan and all, I am evaluating this like you are. You may want to check http://www.tomkleinpeter.com/2008/03/17/programmers-toolbox-part-3-consistent-hashing/. I would appreciate if others can shed some light on this, too. Bests, James On Fri, Sep 10, 2010 at 6:07 AM, Stephan Raemy wrote: > Hi

Re: Date faceting +1MONTH problem

My plan has been to use unix timestamps as integer fields. I also was ooing to use 'all balls' time for dates without time. Midnight is actually AM, so I was going to count it as the next day. To get my range, I was going to use a greater than, and then a less than for the two intgeres, calcula

Re: Solr CoreAdmin create ignores dataDir Parameter

Mark Miller schrieb: I think so - what version of Solr are you using? I believe I've changed this on trunk a few months ago. We are running 1.4.2 and I looked in the solr/tags/release-1.4.1 branch of SVN. The Version in trunk I can see is from 27.07.2010 and this also reads first config a

solr / lucene engineering positions in Boston, MA USA @ the Echo Nest

Hi all, brief message to let you know that we're in heavy hire mode at the Echo Nest. As many of you know we are very heavy solr/lucene users (~1bn documents across many many servers) and a lot of our staff have been working with and contributing to the projects over the years. We are a "music inte

How to delete documents from a SOLR cloud / balance the shards in the cloud?

Hi solr-cloud users, I'm currently setting up a solr-cloud/zookeeper instance and so far, everything works out fine. I downloaded the source from the cloud branch yesterday and build it from source. I've got 10 shards distributed across 4 servers and a zookeeper instance. Searching documents with

Re: Solr CoreAdmin create ignores dataDir Parameter

On 9/10/10 7:00 AM, Frank Wesemann wrote: > Hello, > if I am trying to create a new SolrCore based on an extisting one via > the CoreAdmin HTTP API, > > http://localhost:8983/solr/admin/cores?action=CREATE&name=newCore&instanceDir=old_instance&schema=newSchema.xml&dataDir=newdata >

Solr CoreAdmin create ignores dataDir Parameter

Hello, if I am trying to create a new SolrCore based on an extisting one via the CoreAdmin HTTP API, http://localhost:8983/solr/admin/cores?action=CREATE&name=newCore&instanceDir=old_instance&schema=newSchema.xml&dataDir=newdata

Re: Date faceting +1MONTH problem

Just attended a talk at JavaZone (www.javazone.no) by Stephen Colebourne about JSR-310 which will make these kind of operations easier in future JDK, and how Joda-Time goes a great way of enabling it today. I'm not saying it would fix your GAP issue, as it's all about what definition of "month"

Re: Distance sorting with spatial filtering

Since no one has jumped in to give the right syntax- yeah, it's a bug. Please file a JIRA. On Thu, Sep 9, 2010 at 9:44 PM, Scott K wrote: > On Thu, Sep 9, 2010 at 21:00, Lance Norskog wrote: >> I just checked out the trunk, and branch 3.x This query is accepted on both, >> but gives no responses

Re: How to Update Value of One Field of a Document in Index?

Hi Savannah, You can only reindex the entire document; if you only have the ID, then do a search to retrieve the rest of the data, then reindex. This assumes that all of the fields you need to index are stored (so that you can retrieve them) and not just indexed. Liam On Fri, Sep 10, 2010 at 3: