RE: Out of memory

2011-09-16 Thread Rohit
Hi Luis, First to ans you questions, -Do you index at the same time as users launches queries to twitter index? Yes, currently I am using the same server for indexing and querying due to lack of resources to extent to another server. -Are you using the autocommit option? No, I am committing on

RE: Out of memory

2011-09-16 Thread Rohit
Thanks Chris, This makes sense, at any time we show users a trend graph for all the tweets relevant for them in the last 15 days. So I guess keeping a shards for the last 15-20 days data would be a good option and all the other data moved to different shards each with 2 months data. I have no ide

Re: location of solr folder when deploy to servlet container

2011-09-16 Thread Chris Hostetter
: how do i configure the solr folder to specific directory when deploy to : servlet container. https://wiki.apache.org/solr/SolrInstall -Hoss

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-16 Thread Robert Muir
On Fri, Sep 16, 2011 at 6:53 PM, Burton-West, Tom wrote: > Hello, > > The TieredMergePolicy has become the default with Solr 3.3, but the > configuration in the example uses the mergeFactor setting which applys to the > LogByteSizeMergePolicy. > > How is the mergeFactor interpreted by the Tiered

Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure t

Example for Solr TieredMergePolicy configuration

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure th

Example configuring TieredMergePolicy in Solr

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure th

Re: Out of memory

2011-09-16 Thread Luis Cappa Banda
Hello. Facet queries are slower than others specially when you are working with a 69G index. I would like to know more about the context where occurs the Out of memory exception: is it during an indexation? Do you index at the same time as users launches queries to twitter index? Are you using the

Re: Generating large datasets for Solr proof-of-concept

2011-09-16 Thread Chris Hostetter
: Has anyone ever had to create large mock/dummy datasets for test : environments or for POCs/Demos to convince folks that Solr was the : wave of the future? Any tips would be greatly appreciated. I suppose : it sounds a lot like crawling even though it started out as innocent : DIH usage. the be

Re: How to check if replication is running

2011-09-16 Thread Yury Kats
On 9/16/2011 4:58 PM, Brandon Fish wrote: > Hi Yury, > > You could try checking out the details command of the replication handler: > http://slave_host:port/solr/replication?command=details > which has information such as "isReplicating". How reliable is isReplicating? Is it updated on unexpected

soft commits in EmbeddedSolrServer

2011-09-16 Thread Raimon Bosch
Hi all, I'm checking how to do soft commits with the new version of Solr. I'm using EmbeddedSolrServer to add documents to my index. How can I perform a soft commit using this class? Is it possible? Or should I use the trunk? http://wiki.apache.org/solr/NearRealtimeSearch http://lucene.apache.org

solr unicode problem in query page

2011-09-16 Thread Xue-Feng Yang
I just started to test solr in Chinese content. In the admin page, I could use *:* and saw the correct result. However, when I tried some specific search, I came to no results.  I don't know what the problem is. Any thought? By the way, I have a few cores running in English without encoding

Re: How to check if replication is running

2011-09-16 Thread Brandon Fish
Adding missing link to the issue I mentioned: https://issues.apache.org/jira/browse/SOLR-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851462#action_12851462 2011/9/16 Yury Kats > Let's say I'm forcing a replication of a core using "fetchindex" com

Re: How to check if replication is running

2011-09-16 Thread Brandon Fish
Hi Yury, You could try checking out the details command of the replication handler: http://slave_host:port/solr/replication?command=details which has information such as "isReplicating". You could also look at the script attached to this issue which shows a thorough check of a slaves replication

How to check if replication is running

2011-09-16 Thread Yury Kats
Let's say I'm forcing a replication of a core using "fetchindex" command. No new content is being added to the master. I can check whether replication has finished by periodically querying master and slave for their "indexversion" and comparing the two. But what's the best way to check if replica

OOM errors and -XX:OnOutOfMemoryError flag not working on solr?

2011-09-16 Thread Jason Toy
I have solr issues where I keep running out of memory. I am working on solving the memory issues (this will take a long time), but in the meantime, I'm trying to be notified when the error occurs. I saw with the jvm I can pass the -XX:OnOutOfMemoryError= flag and pass a script to run. Every time t

RE: Out of memory

2011-09-16 Thread Chris Hostetter
: Actually I am storing twitter streaming data into the core, so the rate of : index is about 12tweets(docs)/second. The same solr contains 3 other cores ... : . At any given time I dont need data more than past 15 days, unless : someone queries for it explicetly. How can this be a

Re: Get field value in custom searchcomponent (solr 3.3)

2011-09-16 Thread Chris Hostetter
: What is the best way to get a float field value from docID? : I tried the following code but when it runs throws an exception For input : string: "` ??eI" at line float lat = Float.parseFloat(tlat); the most most straight foward level of abstraction to use (if you want to leverage the field ca

Re: Add copyTo Field without re-indexing?

2011-09-16 Thread Luis Cappa
Hello. You can also develop an application by yourself that uses Solrj to retrieve all the documents from your índex, process and add all the new information (fields) desired and the index them into another Solr index. Its easy. Goodbye! El 16/09/2011, a las 17:39, "Olson, Ron" escribió:

Re: Index not getting refreshed

2011-09-16 Thread Chris Hostetter
: I am commiting but not doing replication now. Mine sort order also includes : last login timestamp. the new profiles are being reflected in my SOLR admin : & db. but its not listed on my website. please be *specific* what do you mean when you say "the new profiles are being reflected in my SO

Re: Solr and DateTimes - bug?

2011-09-16 Thread Chris Hostetter
: I'm wondering if it would be possible for you (or someone else) to fix the : issue with years from 1 to 999 being formatted incorrectly, and then creating : a new ticket for the issue with negative years? you're totally right ... i got so frustrated trying to find a "perfect" solution to the e

Re: DIH URLDataSource paginating with $nextUrl & $hasMore

2011-09-16 Thread B B
BTW - ignore the xpath typo, it should read: On Fri, Sep 16, 2011 at 12:59 PM, B B wrote: > Has anyone successfully setup DIH URLDataSource to paginate imports > using  $nextUrl & $hasMore ?: > > http://wiki.apache.org/solr/DataImportHandler#Configuration_in_data-config.xml-1 > > It's not

DIH URLDataSource paginating with $nextUrl & $hasMore

2011-09-16 Thread B B
Has anyone successfully setup DIH URLDataSource to paginate imports using  $nextUrl & $hasMore ?: http://wiki.apache.org/solr/DataImportHandler#Configuration_in_data-config.xml-1 It's not working for me, the data imports on the first page, but no subsequent calls to the data source are processed.

Re: How to set up the schema to avoid NumberFormatException

2011-09-16 Thread Chris Hostetter
: It is pretty obvious from this that the "sdouble" schema fieldtype is : not setup to parse out group-separators from a number. correct. the numeric (and date) field types are all designed to deal with conversion of the canonical string represetantion. : 1) Then my question is which type pf s

How to set up the schema to avoid NumberFormatException

2011-09-16 Thread Pulkit Singhal
Hello Folks, Surprisingly, the value from the following raw data gives me a NFE (Number Format Exception) when running the DIH (Data Import Handler): $1,000.00 The error logs look like: Caused by: org.apache.solr.common.SolrException: Error while creating field 'price{type=sdouble,properties=inde

Re: Document Boost not evaluated when using standard Query Type?

2011-09-16 Thread Chris Hostetter
: I see that in this use case this makes most sense - thanks. : : But why are fieldNorms in general very corse? they are a floating point value encoded into a single byte... https://lucene.apache.org/java/3_4_0/scoring.html#Score%20Boosting https://lucene.apache.org/java/3_4_0/api/core/org/apac

Re: Add copyTo Field without re-indexing?

2011-09-16 Thread Walter Underwood
On Sep 16, 2011, at 8:39 AM, Olson, Ron wrote: > Is there any way to re-parse an existing index, adding the new copyTo field, > without having to basically start all over again with DIH? No. Indexes cannot be modified. You must reinsert the documents with a new schema. wunder -- Walter Underw

Add copyTo Field without re-indexing?

2011-09-16 Thread Olson, Ron
Hi all- I have an 11 gig index that I realize I need to add another field to, but not from the actual query using DIH, but via copyTo. Is there any way to re-parse an existing index, adding the new copyTo field, without having to basically start all over again with DIH? Thanks, Ron DISCLAIME

solr-user@lucene.apache.org

2011-09-16 Thread eks dev
probably stupid question, Which Directory implementation should be the best suited for index mounted on ramfs/tmpfs? I guess plain old FSDirectory, (or mmap/nio?)

Which Solr / Lucene direcotory for ramfs?

2011-09-16 Thread eks dev
probably stupid question, Which Directory implementation should be the best suited for index mounted on ramfs/tmpfs? I guess plain old FSDirectory, (or mmap/nio?)

getting answers starting with a requested string first

2011-09-16 Thread elisabeth benoit
Hello, Iif I have a request with fq=NAME_ANALYZED:tour eiffel and I have different answers like Restaurant la tour Eiffel Hotel la tour Eiffel Tour Eiffel ... Is there a way to get answers with NAME_ANALYZED beginning with "tour Eiffel" first? Thanks, Elisabeth

Re: add quartz like scheduling cabalities to solr-DIH

2011-09-16 Thread vighnesh
thanks iroxxx but how can l add quartz like scheduling to solr dih ,is there any changes required in anyof the configuration files please specify the procedure. -- View this message in context: http://lucene.472066.n3.nabble.com/add-quartz-like-scheduling-cabalities-to-solr-DIH-tp3341141p334179

Re: can we share the same index directory for multiple cores?

2011-09-16 Thread Erick Erickson
Cores are essentially independent indexes. If you look at your directory structure you'll see something like /cores/core0/data/index /cores/core1/data/index Sharing the same index across multiple cores doesn't make sense from a Solr perspective. And the redundancy comment is kind of odd too. If yo

Re: Complex Fields, Indexing & Storing

2011-09-16 Thread Erick Erickson
Let's quickly review the difference between indexed and stored. Indexing a field means you can search on it. Stored means an entirely separate copy of the raw input is placed in a particular file in your index (*.fdt to be exact). These operations are entirely orthogonal. In effect when you sear

Re: Lucene->SOLR transition

2011-09-16 Thread Tomás Fernández Löbbe
Hi Scott, now your queries are going to be created by a QueryParser. you have a couple of options here, most common are LuceneQueryParser, DismaxQueryParser and ExtendedDismaxQueryParser, but there are others. The QueryParser will be creating all those queries you mentiones, for example, if you are

Re: Error IndexOutOfBoundsException

2011-09-16 Thread Erick Erickson
What version of Solr are you using? Do you have any custom code? I know some JIRAs have mentioned, you could try searching the Solr and Lucene JIRAs at: https://issues.apache.org/jira/browse/SOLR But without a lot more background, it's hard to say. You might review: http://wiki.apache.org/solr/Us

Re: NRT and commit behavior

2011-09-16 Thread Erick Erickson
Uhm, you're putting a lot of index into not very much memory. I really think you're going to have to shard your index across several machines to get past this problem. Simply increasing the size of your caches is still limited by the physical memory you're working with. You really have to put a p

Re: Document Boost not evaluated when using standard Query Type?

2011-09-16 Thread Erick Erickson
At a guess because the documents in question are very similar. Best Erick 2011/9/14 Daniel Pötzinger : >> >> fieldNorms are very corse.  In my opinion, if you have a >> "weighting" you want to use to affect score sort, it's better to index >> that weight as a numeric field, and explicitly factor

Nested JSON Object Indexing and Search

2011-09-16 Thread Dnyaneshwar Dewadkar
Dear all I am trying to index a data using json object. For normal single field it is easy to make things work for me. I have a data to be stored in format like Solr Version 3.4, java implementation Document 1: { "id" : "B34KPNJ0998", "project_name_t":"Manual",

Re: add quartz like scheduling cabalities to solr-DIH

2011-09-16 Thread Ahmet Arslan
> i am unable to do the scheduling for solr DIH,and anyone > specifies the > procedure for how to add quartz like scheduling cabalities > to solr -DIH > .Please let it me know is it possible to add quartz-like > scheduling to solr > DIH. I think most of the people use Operating Stytems' cron-job c

Re: Distinct elements in a field

2011-09-16 Thread swiss knife
I could get this number by using group.ngroups=true&group.limit=0 but doing grouping for this seems like an overkill Would you advise using JIRA SOLR-1814 ? - Original Message - From: swiss knife Sent: 09/15/11 12:43 PM To: solr-user@lucene.apache.org Subject: Distinct elements in a

How To Implement Sweet Spot Similarity?

2011-09-16 Thread Pranav Prakash
I was wondering if there is *any* article on the web that provides me with implementation details and some sort of analysis on Sweet Spot Similarity? Google shows me all the JIRA commits and comments but no article about actual implementation. What are the various configs that could be done. What a

Re: Generating large datasets for Solr proof-of-concept

2011-09-16 Thread Toke Eskildsen
On Thu, 2011-09-15 at 22:54 +0200, Pulkit Singhal wrote: > Has anyone ever had to create large mock/dummy datasets for test > environments or for POCs/Demos to convince folks that Solr was the > wave of the future? Yes, but I did it badly. The problem is that real data are not random so any simple

Re: How To Implement Sweet Spot Similarity?

2011-09-16 Thread Ahmet Arslan
> I was wondering if there is *any* > article on the web that provides me with > implementation details and some sort of analysis on Sweet > Spot Similarity? > Google shows me all the JIRA commits and comments but no > article about > actual implementation. What are the various configs that > could

Re: Lucene->SOLR transition

2011-09-16 Thread Federico Fissore
Hi Scott Smith, il 16/09/2011 02:30, ha scritto: I've been using lucene for a number of years. We've now decided to move to SOLR. I have a couple of questions. 1. I'm used to creating Boolean queries, filter queries, term queries, etc. for lucene. Am I right in thinking that for SOL