Setting up priorities

[Sajith Weerakoon]

Hello,

 

I am developing a search application using solr and now I want to increase
the priority of certain fields. How can I do it? Can someone help me out?

 

Thanks,

Regards,

Sajith Vimukthi Weerakoon.

 

Re: Setting up priorities

[Markus Jelsma - Buyways B.V.]

Fieldboosting on index time might help you out:
http://wiki.apache.org/solr/UpdateXmlMessages#head-8315b8028923d028950ff750a57ee22cbf7977c6


-  
Markus Jelsma  Buyways B.V. Tel. 050-3118123
Technisch ArchitectFriesestraatweg 217c Fax. 050-3118124
http://www.buyways.nl  9743 AD GroningenKvK  01074105


On Mon, 2009-03-30 at 16:50 +0530, Sajith Weerakoon wrote:

> Hello,
> 
>  
> 
> I am developing a search application using solr and now I want to increase
> the priority of certain fields. How can I do it? Can someone help me out?
> 
>  
> 
> Thanks,
> 
> Regards,
> 
> Sajith Vimukthi Weerakoon.
> 
>  
> 

Not getting the proper result.

[Radha C.]

Hi,
 
I am having following analyzer set up in schema.xml

 
  
  

 
  
 
I am indexing a database field which contains L.I.C and I am trying to
search the field as follows but getting zero response.
http://localhost:8080/solr/select/?q=LIC
 &debugQuery=on and
http://localhost:8080/solr/select/?q=lic
 &debugQuery=on
 
But it is giving result for  q=L.I.C
 
It is not identifying the L.I.C and lic .  what is the wrong here? can
anyone help me ?
 
Thanks

Re: Not getting the proper result.

[Grant Ingersoll]

The StandardTokenizer splits on punctuation, so L.I.C. is likely  
becoming 'l', 'i', 'c', while LIC -> lic.  One helpful tool is the  
Analysis page on the Solr admin:  http://localhost:8983/solr/admin/analysis.jsp 
 as it can help you figure out what is going on with analysis on both  
the query and indexing side.


HTH,
Grant

On Mar 30, 2009, at 7:50 AM, Radha C. wrote:


Hi,

I am having following analyzer set up in schema.xml
positionIncrementGap="100">


 
 
 

 

I am indexing a database field which contains L.I.C and I am trying to
search the field as follows but getting zero response.
http://localhost:8080/solr/select/?q=LIC
  
&debugQuery=on and

http://localhost:8080/solr/select/?q=lic
  
&debugQuery=on


But it is giving result for  q=L.I.C

It is not identifying the L.I.C and lic .  what is the wrong here? can
anyone help me ?

Thanks

Re: aka Replication Stall

[Jeff Newburn]

I apologize for the delay.  The replication stalling out doesn't happen
daily.  I will paste the thread dump below to try to help.  This is on a
server that is currently locked on replication for a few hours.  Any more
information please let me know. There are no errors in the logs either so
very strange.

Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
Thread Count: current=73, peak=125, daemon=70
Full Thread Dump:
'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.ms
user time=23940.ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at 
org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
a:182)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at 
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
.java:108)
at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
at 
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
22)
at 
org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
808)
at 
org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
)
at 
org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
at 
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
a:224)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at 
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(ScheduledThreadPoolExecutor.java:181)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:205)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:885)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
at java.lang.Thread.run(Thread.java:619)
'pool-16-thread-1' Id=237, WAITING on
lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5
ff3438a, total cpu time=14952560.ms user time=14177150.ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-190' Id=232, TIMED_WAITING on
lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5
bd74a3c, total cpu time=1035270.ms user time=1001520.ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-189' Id=231, TIMED_WAITING on
lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5
bd74a3c, total cpu time=1483590.ms user time=1437890.ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-188' Id=230, TIMED_WAITING on
lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5
bd74a3c, total cpu time=1486880.ms user time=1440520.ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at 
java.util.concurrent.locks.AbstractQueuedSyn

keepOptimizedOnly solr1.4 trunk version 27/03/2009

[sunnyfr]

Hi,

I would like to know more about keepOptimizedOnly, 
My problem is on the slaves's servers it's a bit slow after a replication
and I would like to automatize an optimization after every commit. How can I
do that ? Is it this option keepOptimizedOnly?

Thanks a lot,

-- 
View this message in context: 
http://www.nabble.com/keepOptimizedOnly-solr1.4-trunk-version-27-03-2009-tp22784884p22784884.html
Sent from the Solr - User mailing list archive at Nabble.com.

Times Replicated Since Startup: 109 since yesterday afternoon?

[sunnyfr]

Hi,

Can you explain me more about this replication script in solr 1.4. 
It does work but it always replicate everything from the master so it lost
every cache everything to replicate it.
I don't get really how it works ? 

Thanks a lot,
-- 
View this message in context: 
http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22784943.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Times Replicated Since Startup: 109 since yesterday afternoon?

[Akshay]

Can you post your replicationhandler configuration?

On Mon, Mar 30, 2009 at 8:17 PM, sunnyfr  wrote:

>
> Hi,
>
> Can you explain me more about this replication script in solr 1.4.
> It does work but it always replicate everything from the master so it lost
> every cache everything to replicate it.
> I don't get really how it works ?
>
> Thanks a lot,
> --
> View this message in context:
> http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22784943.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Akshay K. Ukey.

Re: Strange anomaly(?) with string matching in query

[Kurt Nordstrom]

Does anybody have any further suggestions on what I might try in this
situation?  Any tools perhaps that might help me put my finger on Solr's
pulse so I can figure out just what's going on in there at index and query
time?

-Kurt


Kurt Nordstrom wrote:
> 
> Changed the config so that both WordDelimiterFilterFactory settings on
> both index and query use: 
> 
> org.apache.solr.analysis.WordDelimiterFilterFactory
> {generateNumberParts=1, catenateWords=1, generateWordParts=1,
> catenateAll=0, catenateNumbers=1}
> 
> Restarted Solr, reindexed the records.
> 
> Unfortunately, no change in the search results.  It still won't find that
> pesky string.  It seems to be generating the same results as before in the
> analysis page.
> 
> Any other things I might try or diagnostics that might give useful output?   
> 
> -Kurt
> 
> 


-- 
View this message in context: 
http://www.nabble.com/Strange-anomaly%28-%29-with-string-matching-in-query-tp22704639p22785313.html
Sent from the Solr - User mailing list archive at Nabble.com.

DIH DateFormatTransformer ClassCastException

[Giovanni De Stefano]

Hello all,

I have a little problem with DIH and DateFormatTransformer.

My column in the DB contains an Oracle TIMESTAMP; the corresponding field is
of type date in the Solr schema; I specified a DateFormatTransformer for the
entity; the field definition is as follow:



but I keep getting a ClassCastException...

WARNING: transformer threw error
java.lang.ClassCastException: java.sql.Date
 at
org.apache.solr.handler.dataimport.DateFormatTransformer.transformRow(DateFormatTransformer.java:69)
 at
org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransformer(EntityProcessorBase.java:186)
 at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:80)
 at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
 at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
 at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
 at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
 at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
 at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
17:15:59,964 ERROR [STDERR] Mar 30, 2009 5:15:59 PM
org.apache.solr.handler.dataimport.DocBuilder buildDocument
SEVERE: Exception while processing: downloadable document :
SolrInputDocumnt[{}]
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.ClassCastException: java.sql.Date
 at
org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransformer(EntityProcessorBase.java:206)
 at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:80)
 at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
 at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
 at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
 at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
 at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
 at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
Caused by: java.lang.ClassCastException: java.sql.Date
 at
org.apache.solr.handler.dataimport.DateFormatTransformer.transformRow(DateFormatTransformer.java:69)
 at
org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransformer(EntityProcessorBase.java:186)
 ... 7 more

Any idea? How can I fix this?

Thanks a lot!

Giovanni

Re: Times Replicated Since Startup: 109 since yesterday afternoon?

[sunnyfr]

I've about 30 000 docs updated every 20mn.
I just store id and text which is (title description)
my index is about 11G 


-- 
View this message in context: 
http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22785606.html
Sent from the Solr - User mailing list archive at Nabble.com.

Some Kind of Crazy Histogram

[Jeff Newburn]

I have noticed that I can¹t seem to make sense of the histogram.  For every
field the x-axis shows powers of 2 which make no sense for things like brand
name.  Am I looking at it wrong or is it having issues?
-- 
Jeff Newburn
Software Engineer, Zappos.com
jnewb...@zappos.com - 702-943-7562

Re: NullPointerException while performing Merge

[Michael McCandless]

Do you have an index where this exception happens consistently, eg
when you try to optimize?  Can you post that somewhere?

Also, which exact JRE version are you using?

Mike

On Sun, Mar 29, 2009 at 1:28 PM, Sameer Maggon  wrote:
> In our application, we are getting NullPointerExceptions very frequently. It
> seems like it's happening during the merge operation (commit). There are no
> exceptions while adding documents to Solr. We are using Solr 1.3.0. I looked
> around the mailing list, and found that there is a JIRA issue opened for a
> similar bug (Lucene-1374), but it's not exactly the same. Also, my fields
> are not compressed.
>
> Has anyone seem this before?
>
> Below is the stacktrace.
>
> Exception in thread "Lucene Merge Thread #142"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.NullPointerException
>    at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:325)
>    at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:302)
> Caused by: java.lang.NullPointerException
>    at
> org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:179)
>    at
> org.apache.lucene.index.FieldsWriter.addDocument(FieldsWriter.java:268)
>    at
> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:361)
>    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
>    at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4485)
>    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4143)
>    at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:218)
>    at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:274)
>
> Thanks,
> Sameer.
>

How to round solr score ?

[squaro]

Hello,

I would like to cut solr score to 3 or 4 digits .
Indeed I would like to be able to sort by score, then by another critria (
price for example).
So if two docs have score of 1.67989 and 1.6767, I would like to sort them
by price.

Do you have any idea how I could do that ? 
-- 
View this message in context: 
http://www.nabble.com/How-to-round-solr-score---tp22787254p22787254.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

[Grant Ingersoll]

Fergus,

I think the problem may actually be due to something that was  
introduced by a change to Solr's StopFilterFactory and the way it  
loads the stop words set.  See https://issues.apache.org/jira/browse/SOLR-1095


I am in the process of testing it out and will let you know.

-Grant

On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:


Hey Fergus,

Finally got a chance to run your scripts, etc. per the thread:
http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623

I can reproduce your slowdown.

One oddity with rev 643465 is:

On the old version, there is an exception during startup:
Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
   at  
org 
.apache 
.solr 
.handler 
.component.SearchHandler.handleRequestBody(SearchHandler.java:129)
   at  
org 
.apache 
.solr 
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:125)

   at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
   at  
org 
.apache 
.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java: 
50)

   at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
   at java.util.concurrent.FutureTask 
$Sync.innerRun(FutureTask.java:303)

   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at java.util.concurrent.ThreadPoolExecutor 
$Worker.runTask(ThreadPoolExecutor.java:885)
   at java.util.concurrent.ThreadPoolExecutor 
$Worker.run(ThreadPoolExecutor.java:907)

   at java.lang.Thread.run(Thread.java:637)

I see two things in CHANGES.txt that might apply, but I'm not sure:
1. I think commons-csv was upgraded
2. The CSV loader stuff was refactored to share common code

I'm still investigating.

-Grant


--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Re: Times Replicated Since Startup: 109 since yesterday afternoon?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 8:17 PM, sunnyfr  wrote:

>
> Hi,
>
> Can you explain me more about this replication script in solr 1.4.
> It does work but it always replicate everything from the master so it lost
> every cache everything to replicate it.
> I don't get really how it works ?
>

That's the normal behavior of a commit. The caches contain the internal
document ids assigned by Lucene. When you call commit, the caches are thrown
away since the commit changes the doc ids. This is the reason for
autowarming/regenerating the caches.

-- 
Regards,
Shalin Shekhar Mangar.

Re: DIH DateFormatTransformer ClassCastException

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 8:52 PM, Giovanni De Stefano <
giovanni.destef...@gmail.com> wrote:

> Hello all,
>
> I have a little problem with DIH and DateFormatTransformer.
>
> My column in the DB contains an Oracle TIMESTAMP; the corresponding field
> is
> of type date in the Solr schema; I specified a DateFormatTransformer for
> the
> entity; the field definition is as follow:
>
>  dateTimeFormat="-MM-dd'T'hh:mm:ss'+'"/>
>
> but I keep getting a ClassCastException...
>
> WARNING: transformer threw error
> java.lang.ClassCastException: java.sql.Date


The jdbc equivalent of an Oracle timestamp is java.sql.Date which is a
sub-class of java.util.Date. DateFormatTransformer is used to convert
strings to a date object. Therefore, if your solr field is of date type, you
don't need to use DateFormatTransformer at all.


-- 
Regards,
Shalin Shekhar Mangar.

Re: OOM at MultiSegmentReader.norms

[vivek sar]

Thanks Otis and Mike.

I'm indexing total of 9 fields, with 5 having norms turned on. I think
I may not need it and will try use the omitNorms for them.

How do I make use of RAMBuffer in Solr? I couldn't find anything on
this on the Wiki - any pointer?

Thanks,
-vivek

On Sat, Mar 28, 2009 at 1:09 AM, Michael McCandless
 wrote:
> Still, 1024M ought to be enough to load one field's norms (how many
> fields have norms?).  If you do things requiring FieldCache that'll
> also consume RAM.
>
> It's also possible you're hitting this bug (false OOME) in Sun's JRE:
>
>  http://issues.apache.org/jira/browse/LUCENE-1566
>
> Feel free to go vote for it!
>
> Mike
>
> On Fri, Mar 27, 2009 at 10:11 PM, Otis Gospodnetic
>  wrote:
>>
>> That's a tiny heap.  Part of it is used for indexing, too.  And the fact 
>> that your heap is so small shows you are not really making use of that nice 
>> ramBufferSizeMB setting. :)
>>
>> Also, use omitNorms="true" for fields that don't need norms (if their types 
>> don't already do that).
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
>>
>> - Original Message 
>>> From: vivek sar 
>>> To: solr-user@lucene.apache.org
>>> Sent: Friday, March 27, 2009 6:15:59 PM
>>> Subject: OOM at MultiSegmentReader.norms
>>>
>>> Hi,
>>>
>>>    I've index of size 50G (around 100 million documents) and growing -
>>> around 2000 records (1 rec = 500 byes) are being written every second
>>> continuously. If I make any search on this index I get OOM. I'm using
>>> default cache settings (512,512,256) in the solrconfig.xml. The search
>>> is using the admin interface (returning 10 rows) with no sorting,
>>> faceting or highlighting. Max heap size is 1024m.
>>>
>>> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>>         at
>>> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
>>>         at
>>> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
>>>         at 
>>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
>>>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>>>         at
>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>>>         at
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>>>         at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>>>         at
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>>         at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>         at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>         at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>         at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>>         at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>         at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>
>>> What could be the problem?
>>>
>>> Thanks,
>>> -vivek
>>
>>
>

Re: keepOptimizedOnly solr1.4 trunk version 27/03/2009

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 8:14 PM, sunnyfr  wrote:

>
> I would like to know more about keepOptimizedOnly,
> My problem is on the slaves's servers it's a bit slow after a replication
> and I would like to automatize an optimization after every commit. How can
> I
> do that ? Is it this option keepOptimizedOnly?
>
>
No it is not. I don't think there is a way to do automatic optimize in Solr.

Each commit/optimize on Solr/Lucene creates a commit point (a point-in-time
snapshot of the index at the time of commit). We don't need to keep all the
commit points forever otherwise we will run out of space. At least one
commit point (the last commit point) must be kept always. The
SolrDeletionPolicy is the class which identifies the old commit points which
are not useful anymore and should be deleted.

keepOptimizedOnly tells SolrDeletionPolicy to keep only those commit points
which are optimized. Note that it does not make sense to use
keepOptimizedOnly=true if the maxCommitsToKeep=1.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Times Replicated Since Startup: 109 since yesterday afternoon?

[sunnyfr]

So except commit/optimize or replicate with a time poll less often, I can't
change this ???
So replication when you have loads of data updated every 30mn is not
adviced.
Or I must replicate once a day ??? or ..?


Shalin Shekhar Mangar wrote:
> 
> On Mon, Mar 30, 2009 at 8:17 PM, sunnyfr  wrote:
> 
>>
>> Hi,
>>
>> Can you explain me more about this replication script in solr 1.4.
>> It does work but it always replicate everything from the master so it
>> lost
>> every cache everything to replicate it.
>> I don't get really how it works ?
>>
> 
> That's the normal behavior of a commit. The caches contain the internal
> document ids assigned by Lucene. When you call commit, the caches are
> thrown
> away since the commit changes the doc ids. This is the reason for
> autowarming/regenerating the caches.
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Times-Replicated-Since-Startup%3A-109--since-yesterday-afternoon--tp22784943p22787805.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to round solr score ?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 10:04 PM, squaro  wrote:

>
> Hello,
>
> I would like to cut solr score to 3 or 4 digits .
> Indeed I would like to be able to sort by score, then by another critria (
> price for example).
> So if two docs have score of 1.67989 and 1.6767, I would like to sort them
> by price.
>
> Do you have any idea how I could do that ?


I don't there there is an existing way to round them. But it will be a
useful contribution if you can write a function query for rounding.

Look at http://wiki.apache.org/solr/FunctionQuery

-- 
Regards,
Shalin Shekhar Mangar.

Re: Times Replicated Since Startup: 109 since yesterday afternoon?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 10:31 PM, sunnyfr  wrote:

>
> So except commit/optimize or replicate with a time poll less often, I can't
> change this ???
> So replication when you have loads of data updated every 30mn is not
> adviced.
> Or I must replicate once a day ??? or ..?
>
>
Yes, commits are expensive. It is not advised to do them very often since
the caches are thrown away.

However, there is a lot of work going on to make commits cheaper. Expect to
see some improvements in the coming months.

-- 
Regards,
Shalin Shekhar Mangar.

Re: How to round solr score ?

[Grant Ingersoll]

On Mar 30, 2009, at 1:07 PM, Shalin Shekhar Mangar wrote:

On Mon, Mar 30, 2009 at 10:04 PM, squaro   
wrote:




Hello,

I would like to cut solr score to 3 or 4 digits .
Indeed I would like to be able to sort by score, then by another  
critria (

price for example).
So if two docs have score of 1.67989 and 1.6767, I would like to  
sort them

by price.

Do you have any idea how I could do that ?



I don't there there is an existing way to round them. But it will be a
useful contribution if you can write a function query for rounding.

Look at http://wiki.apache.org/solr/FunctionQuery


What did you have in mind, Shalin?It seems to me you would have to  
hook into the HitCollector and/or implement your own sorting  
capability, as the Func Query is just going to allow you to take price  
in as a scoring factor, no?


-Grant

Re: DIH DateFormatTransformer ClassCastException

[Giovanni De Stefano]

Hello Shalin,

thank you for your reply.

Unfortunately I have to perform some kind of transformation otherwise the
exception is:

org.apache.solr.common.SolrException: Invalid Date
String:'2008-07-12T22:00:26+0200'
at org.apache.solr.schema.DateField.parseMath(DateField.java:167)
at org.apache.solr.schema.DateField.toInternal(DateField.java:138)
at org.apache.solr.schema.FieldType.createField(FieldType.java:179)
at org.apache.solr.schema.SchemaField.createField(SchemaField.java:93)
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:243)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:58)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

Any idea?

Thanks,
Giovanni


On Mon, Mar 30, 2009 at 6:52 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Mon, Mar 30, 2009 at 8:52 PM, Giovanni De Stefano <
> giovanni.destef...@gmail.com> wrote:
>
> > Hello all,
> >
> > I have a little problem with DIH and DateFormatTransformer.
> >
> > My column in the DB contains an Oracle TIMESTAMP; the corresponding field
> > is
> > of type date in the Solr schema; I specified a DateFormatTransformer for
> > the
> > entity; the field definition is as follow:
> >
> >  > dateTimeFormat="-MM-dd'T'hh:mm:ss'+'"/>
> >
> > but I keep getting a ClassCastException...
> >
> > WARNING: transformer threw error
> > java.lang.ClassCastException: java.sql.Date
>
>
> The jdbc equivalent of an Oracle timestamp is java.sql.Date which is a
> sub-class of java.util.Date. DateFormatTransformer is used to convert
> strings to a date object. Therefore, if your solr field is of date type,
> you
> don't need to use DateFormatTransformer at all.
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: How to round solr score ?

[Walter Underwood]

I think what you want to do is add in a function query that gives
values in that range.

There is no need to round the scores. That doesn't do anything
but throw away information.

wunder

On 3/30/09 10:07 AM, "Shalin Shekhar Mangar"  wrote:

> On Mon, Mar 30, 2009 at 10:04 PM, squaro  wrote:
> 
>> 
>> Hello,
>> 
>> I would like to cut solr score to 3 or 4 digits .
>> Indeed I would like to be able to sort by score, then by another critria (
>> price for example).
>> So if two docs have score of 1.67989 and 1.6767, I would like to sort them
>> by price.
>> 
>> Do you have any idea how I could do that ?
> 
> 
> I don't there there is an existing way to round them. But it will be a
> useful contribution if you can write a function query for rounding.
> 
> Look at http://wiki.apache.org/solr/FunctionQuery

Re: How to round solr score ?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 10:54 PM, Grant Ingersoll wrote:

>
> I don't there there is an existing way to round them. But it will be a
>> useful contribution if you can write a function query for rounding.
>>
>> Look at http://wiki.apache.org/solr/FunctionQuery
>>
>
> What did you have in mind, Shalin?It seems to me you would have to hook
> into the HitCollector and/or implement your own sorting capability, as the
> Func Query is just going to allow you to take price in as a scoring factor,
> no?
>

Yonik added a way to use the score of a query in function queries with
SOLR-939. Look at the "query" function on the wiki. Some very cool things
are possible now :)

-- 
Regards,
Shalin Shekhar Mangar.

Re: How to round solr score ?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 11:06 PM, Walter Underwood
wrote:

> I think what you want to do is add in a function query that gives
> values in that range.
>

The scale function won't work in this use-case because it will give you a
double in the given range. So you cannot do sort by score and price. For
this use-case you need to scale to an integer value in a discrete range.

-- 
Regards,
Shalin Shekhar Mangar.

Re: How to round solr score ?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 11:07 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

>
> Yonik added a way to use the score of a query in function queries with
> SOLR-939. Look at the "query" function on the wiki. Some very cool things
> are possible now :)
>

Sorry, that should have been SOLR-1046

-- 
Regards,
Shalin Shekhar Mangar.

Re: How to round solr score ?

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 11:10 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Mon, Mar 30, 2009 at 11:06 PM, Walter Underwood  > wrote:
>
>> I think what you want to do is add in a function query that gives
>> values in that range.
>>
>
> The scale function won't work in this use-case because it will give you a
> double in the given range. So you cannot do sort by score and price. For
> this use-case you need to scale to an integer value in a discrete range.
>

Walter -- I think I misinterpreted your response. Sorry about that. You are
indeed right. However, we can do scale(round(score, 2), 1, 10) or we can
create a new scale function as you said.

-- 
Regards,
Shalin Shekhar Mangar.

Re: DIH DateFormatTransformer ClassCastException

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 10:59 PM, Giovanni De Stefano <
giovanni.destef...@gmail.com> wrote:

>
> Unfortunately I have to perform some kind of transformation otherwise the
> exception is:
>
> org.apache.solr.common.SolrException: Invalid Date
> String:'2008-07-12T22:00:26+0200'
>at org.apache.solr.schema.DateField.parseMath(DateField.java:167)
>at org.apache.solr.schema.DateField.toInternal(DateField.java:138)
>at org.apache.solr.schema.FieldType.createField(FieldType.java:179)
>at org.apache.solr.schema.SchemaField.createField(SchemaField.java:93)
>at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:243)


Hmm that is not expected. Which version of Solr are you using?

-- 
Regards,
Shalin Shekhar Mangar.

Re: DIH DateFormatTransformer ClassCastException

[Giovanni De Stefano]

I was using  a nightly build DIH 1.4 from last week, then I upgraded to the
latest nightly build and still the same problem. :-(

I look forward to hearing from you...

P.S. in a previous post I asked something very similar and you replied to me
suggesting to use Solrj for the integration with Solrj and a
DateFormatTransformer for DIH.

In fact I am using both Solrj and DIH because my data come from Java
serialized objects (thus I use Solrj for the import) and from an Oracle DB
(thus I use DIH).

Cheers,
Giovanni


On Mon, Mar 30, 2009 at 8:01 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Mon, Mar 30, 2009 at 10:59 PM, Giovanni De Stefano <
> giovanni.destef...@gmail.com> wrote:
>
> >
> > Unfortunately I have to perform some kind of transformation otherwise the
> > exception is:
> >
> > org.apache.solr.common.SolrException: Invalid Date
> > String:'2008-07-12T22:00:26+0200'
> >at org.apache.solr.schema.DateField.parseMath(DateField.java:167)
> >at org.apache.solr.schema.DateField.toInternal(DateField.java:138)
> >at org.apache.solr.schema.FieldType.createField(FieldType.java:179)
> >at org.apache.solr.schema.SchemaField.createField(SchemaField.java:93)
> >at
> >
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:243)
>
>
> Hmm that is not expected. Which version of Solr are you using?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: OOM at MultiSegmentReader.norms

[Yonik Seeley]

On Mon, Mar 30, 2009 at 12:53 PM, vivek sar  wrote:
> I'm indexing total of 9 fields, with 5 having norms turned on.

So that's 500MB for norms alone, plus memory for Lucene's term index
(every 128th term by default).  Solr also opens a new
IndexReader/Searcher before closing the old one, so there are no
"outages"... a period when the old searcher has been closed but a new
searcher isn't yet "ready".


-Yonik
http://www.lucidimagination.com

Spelling Component

[Anoop Atre]

Hi all, I'm new to Solr and have been reading up as much as possible,
especially about implementing the spellcheck feature.

The instructions I'm looking at are on the following links:
> http://wiki.apache.org/solr/SpellCheckComponent
> http://snipurl.com/elxmr [vufind tech list]

Error I get when I try to build the index using
$SOLR_URL/solr/biblio/spellCheckCompRH?q=*:*&spellcheck.q=hell&spellcheck=true&spellcheck.build=true

  NOT_FOUND
  RequestURI=/solr/biblio/spellCheckCompRH
  Powered by Jetty://

I've tried it with a separate request handler "/spellCheckCompRH" and
also integrating it into the regular one "/search" but it just doesn't
want to work. Do I need to restart something after changes to
schema/solrconfig files? I looked through the docs and mailing lists but
couldn't figure out what I'm doing wrong and I don't think any restart
is required.

I've attached my schema & solrconfig hoping it's useful...any help will
be appreciated! : )

-- 

- - - - - - - - - - - - - - - - - - - - - - - - -
Anoop Atre
IS Developer & Integrator, MnPALS
PH: 507.389.5060
OF: 3022 Memorial Library (Office-ML 3022)
--
"Mit der Dummheit kämpfen Götter selbst vergebens"
 ~ Johann Christoph Friedrich von Schiller


  








  
	



  







  

  
 
   
   
   
   
   
   
	
 
 id
 text
 


 
	
 
 

  
  
  

 
   explicit
   
 
  

  

  manu,cat
  1

  

  
 
  

  explicit




  spellcheck
  elevator

  

  

  
  textSpell
  true
  true


  default
  spelling
  0.7
  ./spellchecker


  jarowinkler
  spelling
  
  org.apache.lucene.search.spell.JaroWinklerDistance
  ./spellchecker_example
  true


  

  

  


true


true

false

1
true
	
	spelling


spellcheck

  

  

string
elevate.xml
  
 
  

  explicit


  elevator

  

Lower-cased string fieldtype?

[Walter Underwood]

I need a lower-cased string fieldtype (to use with the ElevationComponent),
but I'm temporarily stuck. The LowerCaseFilter needs a tokenizer, but I
don't want to tokenize. What am I missing?

wunder

Re: Lower-cased string fieldtype?

[Shalin Shekhar Mangar]

On Tue, Mar 31, 2009 at 12:38 AM, Walter Underwood
wrote:

> I need a lower-cased string fieldtype (to use with the ElevationComponent),
> but I'm temporarily stuck. The LowerCaseFilter needs a tokenizer, but I
> don't want to tokenize. What am I missing?
>

KeywordTokenizer can help.

-- 
Regards,
Shalin Shekhar Mangar.

RE: Lower-cased string fieldtype?

[Vauthrin, Laurent]

I'm not sure if I did this the right way but here's what I ended up
using:








According to the comments:



Note: I had to use the TextField instead of StrField. 

-Original Message-
From:
solr-user-return-20238-laurent.vauthrin=disney@lucene.apache.org
[mailto:solr-user-return-20238-laurent.vauthrin=disney@lucene.apache
.org] On Behalf Of Walter Underwood
Sent: Monday, March 30, 2009 12:08 PM
To: solr-user@lucene.apache.org
Subject: Lower-cased string fieldtype?

I need a lower-cased string fieldtype (to use with the
ElevationComponent),
but I'm temporarily stuck. The LowerCaseFilter needs a tokenizer, but I
don't want to tokenize. What am I missing?

wunder

Birthday (that's "day" not "date") search query?

[Development Team]

Hi everyone,
 I have an index that stores birth-dates, and I would like to search for
anybody whose birth-date is within X days of a certain month/day. For
example, I'd like to know if anybody's birthday is coming up within a
certain number of days, regardless of what year they were born. How would I
do this using Solr?
 As a follow-up, assuming this query is executed very often, should I
maybe be indexing something other than the birth-date? Such as just the
month-day pair? What is the most efficient way to do such a query?

Sincerely,

 Daryl.

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

[Grant Ingersoll]

Fregus,

Is rev 643465 the absolute latest you tried that still performs?  i.e.  
every revision after is slower?


-Grant

On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:


Fergus,

I think the problem may actually be due to something that was  
introduced by a change to Solr's StopFilterFactory and the way it  
loads the stop words set.  See https://issues.apache.org/jira/browse/SOLR-1095


I am in the process of testing it out and will let you know.

-Grant

On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:


Hey Fergus,

Finally got a chance to run your scripts, etc. per the thread:
http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623

I can reproduce your slowdown.

One oddity with rev 643465 is:

On the old version, there is an exception during startup:
Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
  at  
org 
.apache 
.solr 
.handler 
.component.SearchHandler.handleRequestBody(SearchHandler.java:129)
  at  
org 
.apache 
.solr 
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: 
125)

  at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
  at  
org 
.apache 
.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java: 
50)

  at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
  at java.util.concurrent.FutureTask 
$Sync.innerRun(FutureTask.java:303)

  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at java.util.concurrent.ThreadPoolExecutor 
$Worker.runTask(ThreadPoolExecutor.java:885)
  at java.util.concurrent.ThreadPoolExecutor 
$Worker.run(ThreadPoolExecutor.java:907)

  at java.lang.Thread.run(Thread.java:637)

I see two things in CHANGES.txt that might apply, but I'm not sure:
1. I think commons-csv was upgraded
2. The CSV loader stuff was refactored to share common code

I'm still investigating.

-Grant


--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Re: Lower-cased string fieldtype?

[Walter Underwood]

Thanks! I posted, went to get a sandwich, and came back to see two
answers. The change works, and I can get the fix into production today.
This list is magic.

wunder

On 3/30/09 12:13 PM, "Vauthrin, Laurent" 
wrote:

> I'm not sure if I did this the right way but here's what I ended up
> using:
> 
>  omitNorms="true">
> 
> 
> 
> 
> 
> 
> According to the comments:
> 
> 
> 
> Note: I had to use the TextField instead of StrField.
> 
> -Original Message-
> From:
> solr-user-return-20238-laurent.vauthrin=disney@lucene.apache.org
> [mailto:solr-user-return-20238-laurent.vauthrin=disney@lucene.apache
> .org] On Behalf Of Walter Underwood
> Sent: Monday, March 30, 2009 12:08 PM
> To: solr-user@lucene.apache.org
> Subject: Lower-cased string fieldtype?
> 
> I need a lower-cased string fieldtype (to use with the
> ElevationComponent),
> but I'm temporarily stuck. The LowerCaseFilter needs a tokenizer, but I
> don't want to tokenize. What am I missing?
> 
> wunder
> 

Re: DIH DateFormatTransformer ClassCastException

[Shalin Shekhar Mangar]

On Mon, Mar 30, 2009 at 11:39 PM, Giovanni De Stefano <
giovanni.destef...@gmail.com> wrote:

> I was using  a nightly build DIH 1.4 from last week, then I upgraded to the
> latest nightly build and still the same problem. :-(
>
>
I remembered an old email related to Oracle timestamps:

http://www.lucidimagination.com/search/document/4a2fe28f099c3f15/custom_transformer_to_handle_timestamp

But now I'm confused.

1. The initial stack trace has a ClassCastException with java.sql.Date when
used with DateFormatTransformer. That is expected because Jdbc will give
that type if the database field is of date type. Such values can be indexed
without the need for any conversion.
2. However, when you tried without DateFormatTransformer, the error seems to
suggest that the return type of the row is not java.sql.Date because it is
being converted into a string and then DateField complains that it is not a
valid format.
3. The older email I mentioned above links to the javadoc page which
suggests that the oracle timestamp does not inherit from java.sql.Date

Three conflicts!

I don't have an oracle instance to experiment with. Can you please provide
stack trace on the last nightly build or trunk with DateFormatTransformer
and without? You can also try the workaround I had suggested in the email I
mentioned above to see if that solves the problem.

-- 
Regards,
Shalin Shekhar Mangar.

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

[Grant Ingersoll]

Can you verify that rev 701485 still performs reasonably well?  This  
is from October 2008 and I get similar results to the earlier rev. 
Am now trying some other versions between October and when you first  
reported the issue in November.


-Grant

On Mar 30, 2009, at 3:37 PM, Grant Ingersoll wrote:


Fregus,

Is rev 643465 the absolute latest you tried that still performs?   
i.e. every revision after is slower?


-Grant

On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:


Fergus,

I think the problem may actually be due to something that was  
introduced by a change to Solr's StopFilterFactory and the way it  
loads the stop words set.  See https://issues.apache.org/jira/browse/SOLR-1095


I am in the process of testing it out and will let you know.

-Grant

On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:


Hey Fergus,

Finally got a chance to run your scripts, etc. per the thread:
http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623

I can reproduce your slowdown.

One oddity with rev 643465 is:

On the old version, there is an exception during startup:
Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
 at  
org 
.apache 
.solr 
.handler 
.component.SearchHandler.handleRequestBody(SearchHandler.java:129)
 at  
org 
.apache 
.solr 
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: 
125)

 at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
 at  
org 
.apache 
.solr 
.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)

 at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
 at java.util.concurrent.FutureTask 
$Sync.innerRun(FutureTask.java:303)

 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at java.util.concurrent.ThreadPoolExecutor 
$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor 
$Worker.run(ThreadPoolExecutor.java:907)

 at java.lang.Thread.run(Thread.java:637)

I see two things in CHANGES.txt that might apply, but I'm not sure:
1. I think commons-csv was upgraded
2. The CSV loader stuff was refactored to share common code

I'm still investigating.

-Grant


--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Hijacking Search Requests

[Alex Wang]

Hi everyone,

We have a web application that queries a Solr server through http. What we 
would like to do is to customize the Solr server and hijack the search request. 
If the user search term matches certain rules, then redirect the user to a 
different page without even performing any search in Solr, if the search term 
does not match any rules, then perform search as usual.

The question is how this can be achieved with the new SearchComponent 
architecture.

Any inputs would be appreciated!

Alex

Re: Hijacking Search Requests

[Grant Ingersoll]

I think this can be done with a load balancer such that you don't even  
need to go to Solr, right?  Or, do you mean you want different  
"results" from Solr itself?


On Mar 30, 2009, at 4:31 PM, Alex Wang wrote:


Hi everyone,

We have a web application that queries a Solr server through http.  
What we would like to do is to customize the Solr server and hijack  
the search request. If the user search term matches certain rules,  
then redirect the user to a different page without even performing  
any search in Solr, if the search term does not match any rules,  
then perform search as usual.


The question is how this can be achieved with the new  
SearchComponent architecture.


Any inputs would be appreciated!

Alex


--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

[Fergus McMenemie]

Grant,

After all my playing about at boot camp, I gave things a rest. It
was not till months later that got back to looking at solr again.
So after 643465 (2008-Apr-01)  the next version I tried was 694377 
from (2008-Sep-11). Nothing in between. Yep so 643465 is the latest
version I tried that still performs. Every later revision is slower.

However I need to repeat the tests using 643465, 694377 and whatever
is the latest version. On my macbook I am only seeing a 2x slowdown
of 643465 vis today, where as I had been seeing a 3x slowdown using
my Imac.

Fergus


>Fregus,
>
>Is rev 643465 the absolute latest you tried that still performs?  i.e.  
>every revision after is slower?
>
>-Grant
>
>On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>
>> Fergus,
>>
>> I think the problem may actually be due to something that was  
>> introduced by a change to Solr's StopFilterFactory and the way it  
>> loads the stop words set.  See 
>> https://issues.apache.org/jira/browse/SOLR-1095
>>
>> I am in the process of testing it out and will let you know.
>>
>> -Grant
>>
>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>
>>> Hey Fergus,
>>>
>>> Finally got a chance to run your scripts, etc. per the thread:
>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>
>>> I can reproduce your slowdown.
>>>
>>> One oddity with rev 643465 is:
>>>
>>> On the old version, there is an exception during startup:
>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.NullPointerException
>>>   at  
>>> org 
>>> .apache 
>>> .solr 
>>> .handler 
>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>>   at  
>>> org 
>>> .apache 
>>> .solr 
>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: 
>>> 125)
>>>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>>   at  
>>> org 
>>> .apache 
>>> .solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java: 
>>> 50)
>>>   at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>>   at java.util.concurrent.FutureTask 
>>> $Sync.innerRun(FutureTask.java:303)
>>>   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>   at java.util.concurrent.ThreadPoolExecutor 
>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>>   at java.util.concurrent.ThreadPoolExecutor 
>>> $Worker.run(ThreadPoolExecutor.java:907)
>>>   at java.lang.Thread.run(Thread.java:637)
>>>
>>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>>> 1. I think commons-csv was upgraded
>>> 2. The CSV loader stuff was refactored to share common code
>>>
>>> I'm still investigating.
>>>
>>> -Grant
>>
>> --
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search

-- 

===
Fergus McMenemie   Email:fer...@twig.me.uk
Techmore Ltd   Phone:(UK) 07721 376021

Unix/Mac/Intranets Analyst Programmer
===

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

[Fergus McMenemie]

>Can you verify that rev 701485 still performs reasonably well?  This  
>is from October 2008 and I get similar results to the earlier rev. 
>Am now trying some other versions between October and when you first  
>reported the issue in November.

OK. Can you tell me how to get a hold of revision 701485. What is the
magic svn line?


>On Mar 30, 2009, at 3:37 PM, Grant Ingersoll wrote:
>
>> Fregus,
>>
>> Is rev 643465 the absolute latest you tried that still performs?   
>> i.e. every revision after is slower?
>>
>> -Grant
>>
>> On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>>
>>> Fergus,
>>>
>>> I think the problem may actually be due to something that was  
>>> introduced by a change to Solr's StopFilterFactory and the way it  
>>> loads the stop words set.  See 
>>> https://issues.apache.org/jira/browse/SOLR-1095
>>>
>>> I am in the process of testing it out and will let you know.
>>>
>>> -Grant
>>>
>>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>>
 Hey Fergus,

 Finally got a chance to run your scripts, etc. per the thread:
 http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623

 I can reproduce your slowdown.

 One oddity with rev 643465 is:

 On the old version, there is an exception during startup:
 Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
 SEVERE: java.lang.NullPointerException
  at  
 org 
 .apache 
 .solr 
 .handler 
 .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
  at  
 org 
 .apache 
 .solr 
 .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: 
 125)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
  at  
 org 
 .apache 
 .solr 
 .core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
  at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
  at java.util.concurrent.FutureTask 
 $Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at java.util.concurrent.ThreadPoolExecutor 
 $Worker.runTask(ThreadPoolExecutor.java:885)
  at java.util.concurrent.ThreadPoolExecutor 
 $Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:637)

 I see two things in CHANGES.txt that might apply, but I'm not sure:
 1. I think commons-csv was upgraded
 2. The CSV loader stuff was refactored to share common code

 I'm still investigating.

 -Grant
>>>
>>> --
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
>>> using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>
>> --
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>
>--
>Grant Ingersoll
>http://www.lucidimagination.com/
>
>Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
>using Solr/Lucene:
>http://www.lucidimagination.com/search

-- 

===
Fergus McMenemie   Email:fer...@twig.me.uk
Techmore Ltd   Phone:(UK) 07721 376021

Unix/Mac/Intranets Analyst Programmer
===

RE: Hijacking Search Requests

[Alex Wang]

Thanks Grant. Yes, I was trying to persuade our architect to do this from the 
web application itself, but he prefers to centralize this functionality on the 
Solr server rather than having all client applications implement their own.

Cheers!

Alex



>-Original Message-
>From: Grant Ingersoll [mailto:gsing...@apache.org]
>Sent: Monday, March 30, 2009 4:49 PM
>To: solr-user@lucene.apache.org
>Subject: Re: Hijacking Search Requests
>
>I think this can be done with a load balancer such that you don't even
>need to go to Solr, right?  Or, do you mean you want different
>"results" from Solr itself?
>
>On Mar 30, 2009, at 4:31 PM, Alex Wang wrote:
>
>> Hi everyone,
>>
>> We have a web application that queries a Solr server through http.
>> What we would like to do is to customize the Solr server and hijack
>> the search request. If the user search term matches certain rules,
>> then redirect the user to a different page without even performing
>> any search in Solr, if the search term does not match any rules,
>> then perform search as usual.
>>
>> The question is how this can be achieved with the new
>> SearchComponent architecture.
>>
>> Any inputs would be appreciated!
>>
>> Alex
>
>--
>Grant Ingersoll
>http://www.lucidimagination.com/
>
>Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>using Solr/Lucene:
>http://www.lucidimagination.com/search

Wildcard searches

[Vauthrin, Laurent]

Hello again,

I'm in the process of converting one of our services that was previously
using Lucene to use Solr instead.  The main focus here is to preserve
backwards compatibility (even if some searches are not as efficient).
There are currently two scenarios that are giving me problems right now.

1. Leading wildcard searches/suffix searches (e.g. *ickey)
I've looked at https://issues.apache.org/jira/browse/SOLR-218.  Is the
best approach to create a QParserPlugin and change the parser to allow
leading wildcards - setAllowLeadingWildcard(true)?  At the moment we're
trying to avoid indexing terms in reverse order.

2. Phrase searches with wildcards (e.g. "Mickey Mou*")
>From what I understand, Solr/Lucene doesn't support this but we used to
get results with the following code:

new WildcardQuery(new Term("U_name", " Mickey Mou*"))

Is it possible for me to allow this capability in a QParserPlugin?  Is
there another way for me to do it?

Thanks,
Laurent Vauthrin

Adding text document

[nga pham]

Hi All,

I am new to Solr.  Can you please tell me, how can I add a text document?

Thank you,
Nga

Querying Different Index Types in a single request

[Todd Benge]

Hi,

We're in the early stages of converting a large scale lucene deployment to
Solr.  The primary reason is to take advantage of the distributed search
capabilities.

We have multiple sets of indices that serve different types of documents.
In our existing deployment, we use MultiSearcher and custom servlets to load
those indices and serve the response in a single request.

An example of the request would be searching for some media type and we'll
respond with the search results with that type as well as suggested other
media of other types that may be applicable.

In looking through the process for sharding data across multiple indices, I
don't see a way to combine results from mutliple cores into a single
response.  It looks like any data that is added to the SolrQueryResponse
object that is not part of the standard "response" key is filtered out
during the merging of documents.

Is there any other way to query multiple index types in a single query
across a cluster?

Thanks for any help.

Todd

Re: Adding text document

[Eric Pugh]

I would work through this tutorial and then ask specific questions: 
http://lucene.apache.org/solr/tutorial.html

Alternatively there are some commercial support options: 
http://wiki.apache.org/solr/Support

Eric

On Mar 30, 2009, at 6:36 PM, nga pham wrote:


Hi All,

I am new to Solr.  Can you please tell me, how can I add a text  
document?


Thank you,
Nga


-
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com
Free/Busy: http://tinyurl.com/eric-cal

Re: DIH DateFormatTransformer ClassCastException

[Noble Paul നോബിള്‍ नोब्ळ्]

Giovanni,
 guess this patch must help. could you plz verify?
--Noble

On Tue, Mar 31, 2009 at 1:36 AM, Shalin Shekhar Mangar
 wrote:
> On Mon, Mar 30, 2009 at 11:39 PM, Giovanni De Stefano <
> giovanni.destef...@gmail.com> wrote:
>
>> I was using  a nightly build DIH 1.4 from last week, then I upgraded to the
>> latest nightly build and still the same problem. :-(
>>
>>
> I remembered an old email related to Oracle timestamps:
>
> http://www.lucidimagination.com/search/document/4a2fe28f099c3f15/custom_transformer_to_handle_timestamp
>
> But now I'm confused.
>
> 1. The initial stack trace has a ClassCastException with java.sql.Date when
> used with DateFormatTransformer. That is expected because Jdbc will give
> that type if the database field is of date type. Such values can be indexed
> without the need for any conversion.
> 2. However, when you tried without DateFormatTransformer, the error seems to
> suggest that the return type of the row is not java.sql.Date because it is
> being converted into a string and then DateField complains that it is not a
> valid format.
> 3. The older email I mentioned above links to the javadoc page which
> suggests that the oracle timestamp does not inherit from java.sql.Date
>
> Three conflicts!
>
> I don't have an oracle instance to experiment with. Can you please provide
> stack trace on the last nightly build or trunk with DateFormatTransformer
> and without? You can also try the workaround I had suggested in the email I
> mentioned above to see if that solves the problem.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
--Noble Paul

Re: Querying Different Index Types in a single request

[Yonik Seeley]

On Mon, Mar 30, 2009 at 7:07 PM, Todd Benge  wrote:
> Is there any other way to query multiple index types in a single query
> across a cluster?

Splitting a homogeneous index into shards, querying across them, and
combining the results is distributed search.  If the shards are
heterogeneous (have different "schemas") then it's closer to federated
search.  Solr's distributed search may work for federated search if
you combine all the schemas into one though.  What information are you
missing from the distributed request that you can get from a single
distributed Solr shard?

-Yonik
http://www.lucidimagination.com

Re: Hijacking Search Requests

[Noble Paul നോബിള്‍ नोब्ळ्]

you may write a servlet filter which is applied before the
SolrDispatchFilter which applies the rules and do the redirect/forward


On Tue, Mar 31, 2009 at 2:54 AM, Alex Wang  wrote:
> Thanks Grant. Yes, I was trying to persuade our architect to do this from the 
> web application itself, but he prefers to centralize this functionality on 
> the Solr server rather than having all client applications implement their 
> own.
>
> Cheers!
>
> Alex
>
>
>
>>-Original Message-
>>From: Grant Ingersoll [mailto:gsing...@apache.org]
>>Sent: Monday, March 30, 2009 4:49 PM
>>To: solr-user@lucene.apache.org
>>Subject: Re: Hijacking Search Requests
>>
>>I think this can be done with a load balancer such that you don't even
>>need to go to Solr, right?  Or, do you mean you want different
>>"results" from Solr itself?
>>
>>On Mar 30, 2009, at 4:31 PM, Alex Wang wrote:
>>
>>> Hi everyone,
>>>
>>> We have a web application that queries a Solr server through http.
>>> What we would like to do is to customize the Solr server and hijack
>>> the search request. If the user search term matches certain rules,
>>> then redirect the user to a different page without even performing
>>> any search in Solr, if the search term does not match any rules,
>>> then perform search as usual.
>>>
>>> The question is how this can be achieved with the new
>>> SearchComponent architecture.
>>>
>>> Any inputs would be appreciated!
>>>
>>> Alex
>>
>>--
>>Grant Ingersoll
>>http://www.lucidimagination.com/
>>
>>Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>using Solr/Lucene:
>>http://www.lucidimagination.com/search
>
>



-- 
--Noble Paul

Issue in Range Search Query

[dabboo]

Hi,

I am searching for all the records in the index where author's last name
starts with a or b. I am giving my query like 

au_last_name:[a* TO b*]

But it is returning all the other records where last name starts with E, P
or other letter.

Please suggest how I can achieve this. I dont want to hit solr 2 times.
Because in future it could be the case, if I have to search for a to z.

Also, is there any way I can get individual count of records starting with
one particular alphabet.

Thanks,
Amit Garg
-- 
View this message in context: 
http://www.nabble.com/Issue-in-Range-Search-Query-tp22798861p22798861.html
Sent from the Solr - User mailing list archive at Nabble.com.