Regarding WordDelimiterFactory

[Sandhya Agarwal]

Hello,

I have a file with the input string "91{40}9490949090", and I wanted to return 
this file when I search for the query string "+91?40?9*".  The problem is that, 
the input string is getting indexed as 3 terms "91", "40", "9490949090".   Is 
there a way to consider "{" and "}" as part of the string itself.  Can we 
configure WordDelimiterFilterFactory *not to consider* curly braces as 
delimiters?

Thanks,
Sandhya

Re: Regarding WordDelimiterFactory

[Robert Muir]

On Thu, Sep 9, 2010 at 3:57 AM, Sandhya Agarwal wrote:

> Hello,
>
> I have a file with the input string "91{40}9490949090", and I wanted to
> return this file when I search for the query string "+91?40?9*".  The
> problem is that, the input string is getting indexed as 3 terms "91", "40",
> "9490949090".   Is there a way to consider "{" and "}" as part of the string
> itself.  Can we configure WordDelimiterFilterFactory *not to consider* curly
> braces as delimiters?
>

See: https://issues.apache.org/jira/browse/SOLR-2059

as a workaround, if you dont want to use trunk, you could also turn on
preserveOriginal

-- 
Robert Muir
rcm...@gmail.com

org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : SingleInstanceLock: write.lock

[Bharat Jain]

Hi,

We are using SOLR 1.3 and getting this error often. There is only one
instance that index the data. I did some analysis which I have put below and
scenario when this error can happen. Can you guys please validate the issue?
thanks a lot in advance.

SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out
: SingleInstanceLock: write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:938)
at
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:116)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHand
ler.java:122)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHa
ndler2.java:167)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
r2.java:221)

I think this error can happen in the following scneario

1. Thread T1 enters the commit method and its actually an optimize request.
a. T1 gets the iwCommit lock
b. T1 enters the openWriter() method
c. T1 does the (writer == null check) -> time x1

2. Thread T2 enters the addDoc method
a. T2 gets iwAccess lock
b. T2 gets the mutex "this"
c. T2 enters operWriter method
d. T2 does the (writer == null check) -> time x1

Now after the 1.c is done thread yields and 2.d gets execution and it also
sees writer as null and so now both threads will try to create the
indexwriter and it will fail. I have pasted the relevant portion of code
here.


  // iwCommit protects internal data and open/close of the IndexWriter and
  // is a mutex. Any use of the index writer should be protected by
iwAccess,
  // which admits multiple simultaneous acquisitions.  iwAccess is
  // mutually-exclusive with the iwCommit lock.
  protected final Lock iwAccess, iwCommit;



// must only be called when iwCommit lock held
  protected void openWriter() throws IOException {
if (writer==null) {
  writer = createMainIndexWriter("DirectUpdateHandler2", false);
}
  }

addDoc(...) {
...

iwAccess.lock();
try {

  // We can't use iwCommit to protect internal data here, since it would
  // block other addDoc calls.  Hence, we synchronize to protect
internal
  // state.  This is safe as all other state-changing operations are
  // protected with iwCommit (which iwAccess excludes from this block).
  synchronized (this) {
// adding document -- prep writer
closeSearcher();
openWriter();
tracker.addedDocument();
  } // end synchronized block

  // this is the only unsynchronized code in the iwAccess block, which
  // should account for most of the time

}

commit() {
...

iwCommit.lock();
try {
  log.info("start "+cmd);

  if (cmd.optimize) {
closeSearcher();
openWriter();
writer.optimize(cmd.maxOptimizeSegments);
  }

  closeSearcher();
  closeWriter();

  callPostCommitCallbacks();
}

Thanks
Bharat Jain

Re: Regarding WordDelimiterFactory

[Grijesh.singh]

set splitWordsPart=0,splitNumberPart=0 

-
Grijesh
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Regarding-WordDelimiterFactory-tp1444694p1444742.html
Sent from the Solr - User mailing list archive at Nabble.com.

svn branch issues

[Mark Allan]

Hi all,

As I've mentioned in the past, I've created some custom field types  
which make use of the AbstractSubTypeFieldType class in the current  
trunk version of solr for a service we're working on.  We're getting  
close to putting our service into production (early 2011) and we're  
now looking for a stable version of Solr to use with these classes.   
Unfortunately, my field types don't compile against the current stable  
version (Solr 1.4) because of the missing AbstractSubTypeFieldType and  
other required classes.


Having had a look at JIRA to see the number of outstanding unresolved  
issues, I tried downloading the now defunct 1.5 branch on the  
assumption that it's more stable than the current trunk.  Whether or  
not that's a safe assumption remains to be seen!


Anyway, the problem is when I try to checkout the 1.5 branch, I get an  
error from subversion:


$ svn co 
http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
	svn: Repository moved permanently to '/viewvc/lucene/solr/branches/ 
branch-1.5-dev/'; please relocate


Going to http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev 
 in a browser shows the web view and contents of that branch, so  
something's not right with the subversion server.


Anyone got any pointers please?

Alternatively, how stable is the current trunk? Does it have a long  
way to go before being released as a stable version?


Many thanks
Mark

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: Indexing data on MSSQL failed: Caused by: org.apache.solr.common.SolrException: Error loading class 'com.micros oft.sqlserver.jdbc.SQLServerDriver'

[alokdayal]

Hi Peter,

I've also faced the same problem which you did. I also changed my
data-config.xml file same as yours and checked that my MS Sql Server is
running on default port i.e, 3306 given the same to connection string but
still its not working.

Could you pls. share your solrconfig.xml and schema.xml file so that I can
change it accordingly and my problem would be solved

Thanks 

ALOK  
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Indexing-data-on-MSSQL-failed-Caused-by-org-apache-solr-common-SolrException-Error-loading-class-com-tp1015137p1444820.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: svn branch issues

[Markus Jelsma]

 http://svn.apache.org/repos/asf/lucene/dev/branches/
 
-Original message-
From: Mark Allan 
Sent: Thu 09-09-2010 10:44
To: solr-user@lucene.apache.org; 
Subject: svn branch issues

Hi all,

As I've mentioned in the past, I've created some custom field types  
which make use of the AbstractSubTypeFieldType class in the current  
trunk version of solr for a service we're working on.  We're getting  
close to putting our service into production (early 2011) and we're  
now looking for a stable version of Solr to use with these classes.   
Unfortunately, my field types don't compile against the current stable  
version (Solr 1.4) because of the missing AbstractSubTypeFieldType and  
other required classes.

Having had a look at JIRA to see the number of outstanding unresolved  
issues, I tried downloading the now defunct 1.5 branch on the  
assumption that it's more stable than the current trunk.  Whether or  
not that's a safe assumption remains to be seen!

Anyway, the problem is when I try to checkout the 1.5 branch, I get an  
error from subversion:

$ svn co http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
svn: Repository moved permanently to '/viewvc/lucene/solr/branches/ 
branch-1.5-dev/'; please relocate

Going to http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev 
 in a browser shows the web view and contents of that branch, so  
something's not right with the subversion server.

Anyone got any pointers please?

Alternatively, how stable is the current trunk? Does it have a long  
way to go before being released as a stable version?

Many thanks
Mark

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: Batch update, order of evaluation

[Greg Pendlebury]

I can't reproduce reliably, so I'm suspecting there are issues in our code.
I'm refactoring to avoid the problem entirely.

Thanks for the response though Erick.

Greg

On 8 September 2010 21:51, Greg Pendlebury wrote:

> Thanks,
>
> I'll create a deliberate test tomorrow feed some random data through it
> several times to see what happens.
>
> I'm also working on simply improving the buffer to handle the situation
> internally, but a few hours of testing isn't a big deal.
>
> Ta,
> Greg
>
>
> On 8 September 2010 21:41, Erick Erickson  wrote:
>
>> This would be surprising behavior, if you can reliably reproduce this
>> it's worth a JIRA.
>>
>> But (and I'm stretching a bit here) are you sure you're committing at the
>> end of the batch AND are you sure you're looking after the commit? Here's
>> the scenario: Your updated document is a position 1 and 100 in your batch.
>> Somewhere around SOLR processing document 50, an autocommit occurs,
>> and you're looking at your results before SOLR gets around to committing
>> document 100. Like I said, it's a stretch.
>>
>> To test this, you need to be absolutely sure of two things before you
>> search:
>> 1> the batch is finished processing
>> 2> you've issued a commit after the last document in the batch.
>>
>> If you're sure of the above and still see the problem, please let us
>> know...
>>
>> HTH
>> Erick
>>
>> On Tue, Sep 7, 2010 at 10:32 PM, Greg Pendlebury
>> wrote:
>>
>> > Does anyone know with certainty how (or even if) order is evaluated when
>> > updates are performed by batch?
>> >
>> > Our application internally buffers solr documents for speed of ingest
>> > before
>> > sending them to the server in chunks. The XML documents sent to the solr
>> > server contain all documents in the order they arrived without any
>> settings
>> > changed from the defaults (so overwrite = true). We are careful to avoid
>> > things like HashMaps on our side since they'd lose the order, but I
>> can't
>> > be
>> > certain what occurs inside Solr.
>> >
>> > Sometimes if an object has been indexed twice for various reasons it
>> could
>> > appear twice in the buffer but the most up-to-date version is always
>> last.
>> > I
>> > have however observed instances where the first copy of the document is
>> > indexed and differences in the second copy are missing. Does this sound
>> > likely? And if so are there any obvious settings I can play with to get
>> the
>> > behavior I desire?
>> >
>> > I looked at:
>> > http://wiki.apache.org/solr/UpdateXmlMessages
>> >
>> > but there is no mention of order, just the overwrite flag (which I'm
>> unsure
>> > how it is applied internally to an update message) and the deprecated
>> > duplicates flag (which I have no idea about).
>> >
>> > Would switching to SolrInputDocuments on a CommonsHttpSolrServer help?
>> as
>> > per http://wiki.apache.org/solr/Solrj. This is no mention of order
>> there
>> > either however.
>> >
>> > Thanks to anyone who took the time to read this.
>> >
>> > Ta,
>> > Greg
>> >
>>
>
>

Re: svn branch issues

[Mark Allan]

Thanks. Are you suggesting I use branch_3x and is that considered  
stable?

Cheers
Mark

On 9 Sep 2010, at 10:47 am, Markus Jelsma wrote:


 http://svn.apache.org/repos/asf/lucene/dev/branches/

-Original message-
From: Mark Allan 
Sent: Thu 09-09-2010 10:44
To: solr-user@lucene.apache.org;
Subject: svn branch issues

Hi all,

As I've mentioned in the past, I've created some custom field types
which make use of the AbstractSubTypeFieldType class in the current
trunk version of solr for a service we're working on.  We're getting
close to putting our service into production (early 2011) and we're
now looking for a stable version of Solr to use with these classes.
Unfortunately, my field types don't compile against the current stable
version (Solr 1.4) because of the missing AbstractSubTypeFieldType and
other required classes.

Having had a look at JIRA to see the number of outstanding unresolved
issues, I tried downloading the now defunct 1.5 branch on the
assumption that it's more stable than the current trunk.  Whether or
not that's a safe assumption remains to be seen!

Anyway, the problem is when I try to checkout the 1.5 branch, I get an
error from subversion:

$ svn co http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
svn: Repository moved permanently to '/viewvc/lucene/solr/branches/
branch-1.5-dev/'; please relocate

Going to http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
 in a browser shows the web view and contents of that branch, so
something's not right with the subversion server.

Anyone got any pointers please?

Alternatively, how stable is the current trunk? Does it have a long
way to go before being released as a stable version?

Many thanks
Mark



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

notice: index corruption on Solr/Lucene trunk/3.x branch

[Robert Muir]

If you are using SimpleFSDirectory (either explicitly or via
FSDirectory.open on Windows) with Solr/Lucene trunk or 3.x branch since July
30,
you might have index corruption and you should svn up and rebuild.

More details available here:
https://issues.apache.org/jira/browse/LUCENE-2637

-- 
Robert Muir
rcm...@gmail.com

Re: svn branch issues

[Markus Jelsma]

Well, it's under heavy development but the 3.x branch is more likely to become 
released than 1.5.x, which is highly unlikely to be ever released.


On Thursday 09 September 2010 13:04:38 Mark Allan wrote:
> Thanks. Are you suggesting I use branch_3x and is that considered
> stable?
> Cheers
> Mark
> 
> On 9 Sep 2010, at 10:47 am, Markus Jelsma wrote:
> >  http://svn.apache.org/repos/asf/lucene/dev/branches/
> >
> > -Original message-
> > From: Mark Allan 
> > Sent: Thu 09-09-2010 10:44
> > To: solr-user@lucene.apache.org;
> > Subject: svn branch issues
> >
> > Hi all,
> >
> > As I've mentioned in the past, I've created some custom field types
> > which make use of the AbstractSubTypeFieldType class in the current
> > trunk version of solr for a service we're working on.  We're getting
> > close to putting our service into production (early 2011) and we're
> > now looking for a stable version of Solr to use with these classes.
> > Unfortunately, my field types don't compile against the current stable
> > version (Solr 1.4) because of the missing AbstractSubTypeFieldType and
> > other required classes.
> >
> > Having had a look at JIRA to see the number of outstanding unresolved
> > issues, I tried downloading the now defunct 1.5 branch on the
> > assumption that it's more stable than the current trunk.  Whether or
> > not that's a safe assumption remains to be seen!
> >
> > Anyway, the problem is when I try to checkout the 1.5 branch, I get an
> > error from subversion:
> >
> > $ svn co http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
> > svn: Repository moved permanently to '/viewvc/lucene/solr/branches/
> > branch-1.5-dev/'; please relocate
> >
> > Going to http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
> >  in a browser shows the web view and contents of that branch, so
> > something's not right with the subversion server.
> >
> > Anyone got any pointers please?
> >
> > Alternatively, how stable is the current trunk? Does it have a long
> > way to go before being released as a stable version?
> >
> > Many thanks
> > Mark
> 

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Problem in using solrJs

[ankita shinde]

Hello,
   I am using solrJs. I am trying to do Reuters example. I have followed
all the steps but it's giving error in error console of browser that
"AjaxSolr is undefined".
   I have all the jar files that are required.
So tell me what should i do? Can you give me simple example of solrJS for
sending query to solr and getting response back.
Thank you.

Indexing checksum of field value

[Staffan]

Hi,

I am looking for a way to store the checksum of a field's value, something like:




...


I haven't found anything like that in the docs or on google. Did I
miss something? If not, would a custom tokenizer be a good way to
implement it?

/Staffan

Re: Indexing checksum of field value

[Markus Jelsma]

Hi,

You can use an UpdateProcessor to do so. This can be used to deduplicate 
documents based on exact or near matches with fields in other documents. Check 
the wiki page on deduplication [1] for an example.

[1]: http://wiki.apache.org/solr/Deduplication

Cheers,

On Thursday 09 September 2010 13:44:55 Staffan wrote:
> Hi,
> 
> I am looking for a way to store the checksum of a field's value, something
>  like:
> 
> 
> 
> 
> ...
> 
> 
> I haven't found anything like that in the docs or on google. Did I
> miss something? If not, would a custom tokenizer be a good way to
> implement it?
> 
> /Staffan
> 

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Garbled facets even in a zero hit search

[Dennis Schafroth]

Hi, Running on a Debian 5.0.5 64bit box. Usingsolr-1.4.1 with Java version "1.6.0_20"I am seeing weird facets results along with the "right" looking ones. Garbled data, stuff that looks like a buffer overflow / index off by ...And I even get them when I do a zero hit search. I wouldn't expect any facets:  version="1.0" encoding="UTF-8"?>      0int>    56int>          truestr>      satay:8985/solrstr>      0str>      title:xzyzxstr>      10str>      10str>              author_exactstr>        datestr>        subject_exactstr>      arr>      10str>      20str>    lst>  lst>                          0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>      lst>              0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>      lst>              0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>        0int>      lst>    lst>      lst>response>

response_formated.xml
Description: XML document
I tried to look for a bug report, but haven't been able to find one that matches. I will try to setup a debug session to get closer, but would love to get feedback if this is a know issue.cheers, :-Dennis Schafroth 

how to normalize a query

[dan sutton]

Hi,

Does anyone know how I might normalized a query so that e.g. q="one two"
equals q="two one"

Cheers,
Dan

Re: Garbled facets even in a zero hit search

[Erick Erickson]

That looks...er...unfortunate. The very first thing I'd do is check your
index
and see if there are such weird values in your facet fields. My guess is
that SOLR is working fine, but you somehow have garbage values
in your index, but that's only a guess. I'd try that before trying to
debug, GIGO.

Which wouldn't answer the question of how the garbage got in there
in the first place, posting your field type definition for your
faceted fields would help with that question.

Best
Erick

On Thu, Sep 9, 2010 at 8:19 AM, Dennis Schafroth wrote:

> Hi,
>
> Running on a Debian 5.0.5 64bit box. Using
> solr-1.4.1 with Java version "1.6.0_20"
>
> I am seeing weird facets results along with the "right" looking ones.
> Garbled data, stuff that looks like a buffer overflow / index off by ...
>
> And I even get them when I do a zero hit search. I wouldn't expect any
> facets:
>
> 
> 
>   
> 0
> 56
> 
>   true
>   satay:8985/solr
>   0
>   title:xzyzx
>   10
>   10
>   
> author_exact
> date
> subject_exact
>   
>   10
>   20
> 
>   
>   
>   
> 
> 
>   
> 0
> 0
> 0 int>
> 0
> 0
> 0
> 0
> 0
> 0
> 0
>   
>   
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
>   
>   
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
>   
> 
> 
>   
> 
>
>
> I tried to look for a bug report, but haven't been able to find one that 
> matches. I will try to setup a debug session to get closer, but would love to 
> get feedback if this is a know issue.
>
> cheers,
>
> :-Dennis Schafroth
>
>
>

Inconsistent search results with multiple keywords

[Stéphane Corlosquet]

Hi all,

I'm new to solr so please let me know if there is a more appropriate place
for my question below.

I'm noticing a rather unexpected number of results when I add more keywords
to a search. I'm listing below a example (where I replaced the real keywords
with placeholders):

keyword1 851 hits
keyword1 keyword2  90 hits
keyword1 keyword2 keyword3 269 hits
keyword1 keyword2 keyword3 keyword4 47 hits

As you can see, adding k2 narrows down the amount of results (as I would
expect), but adding k3 to k1 and k2 suddenly increases the amount of
results. with 4 keywords, the results have been narrowed down again. Would
solr/lucene search algorithm with multiple keywords explain this non
consistent behavior? I would think that adding more keywords would narrow
down my results.

I'm pasting below the relevant log in case it helps:

INFO: [] webapp=/solr path=/select/
params={spellcheck=true&facet=true&facet.mincount=1&facet.limit=20&spellcheck.q=keyword1+keyword2+keyword3+keyword4&
json.nl=map&wt=json&version=1.2&rows=10&fl=id,nid,title,comment_count,type,created,changed,score,path,url,uid,name&start=0&facet.sort=true&q=keyword1+keyword2+keyword3+keyword4&bf=recip(rord(created),4,10704,10704)^200.0&facet.field=im_cck_field_author&facet.field=type&facet.field=im_vid_1=&indent=on&start=0&version=2.2&rows=10}
hits=10704 status=0 QTime=1

any hint on whether this is expected or not appreciated.

Steph.

Re: Inconsistent search results with multiple keywords

[Erick Erickson]

Could you show us the  definitions for your fields? I suspect
you're not getting the tokens you expect. This will almost certainly
be true if the type is "string" rather than "text".

The solr admin page (especially analysis) will help you a lot here, as
will adding &debugQuery=on to your query and seeing how the query
is actually processed. Showing us the results of this will also help.

Best
Erick

On Thu, Sep 9, 2010 at 9:34 AM, Stéphane Corlosquet
wrote:

> Hi all,
>
> I'm new to solr so please let me know if there is a more appropriate place
> for my question below.
>
> I'm noticing a rather unexpected number of results when I add more keywords
> to a search. I'm listing below a example (where I replaced the real
> keywords
> with placeholders):
>
> keyword1 851 hits
> keyword1 keyword2  90 hits
> keyword1 keyword2 keyword3 269 hits
> keyword1 keyword2 keyword3 keyword4 47 hits
>
> As you can see, adding k2 narrows down the amount of results (as I would
> expect), but adding k3 to k1 and k2 suddenly increases the amount of
> results. with 4 keywords, the results have been narrowed down again. Would
> solr/lucene search algorithm with multiple keywords explain this non
> consistent behavior? I would think that adding more keywords would narrow
> down my results.
>
> I'm pasting below the relevant log in case it helps:
>
> INFO: [] webapp=/solr path=/select/
>
> params={spellcheck=true&facet=true&facet.mincount=1&facet.limit=20&spellcheck.q=keyword1+keyword2+keyword3+keyword4&
> json.nl
> =map&wt=json&version=1.2&rows=10&fl=id,nid,title,comment_count,type,created,changed,score,path,url,uid,name&start=0&facet.sort=true&q=keyword1+keyword2+keyword3+keyword4&bf=recip(rord(created),4,10704,10704)^200.0&facet.field=im_cck_field_author&facet.field=type&facet.field=im_vid_1=&indent=on&start=0&version=2.2&rows=10}
> hits=10704 status=0 QTime=1
>
> any hint on whether this is expected or not appreciated.
>
> Steph.
>

Re: Inconsistent search results with multiple keywords

[Markus Jelsma]

Looks like AND is your defaultOperator [1]. Check your schema.xml and try 
adding q.op=or to your query.

[1]: http://wiki.apache.org/solr/SearchHandler#q.op
 
On Thursday 09 September 2010 15:34:52 Stéphane Corlosquet wrote:
> Hi all,
> 
> I'm new to solr so please let me know if there is a more appropriate place
> for my question below.
> 
> I'm noticing a rather unexpected number of results when I add more keywords
> to a search. I'm listing below a example (where I replaced the real
>  keywords with placeholders):
> 
> keyword1 851 hits
> keyword1 keyword2  90 hits
> keyword1 keyword2 keyword3 269 hits
> keyword1 keyword2 keyword3 keyword4 47 hits
> 
> As you can see, adding k2 narrows down the amount of results (as I would
> expect), but adding k3 to k1 and k2 suddenly increases the amount of
> results. with 4 keywords, the results have been narrowed down again. Would
> solr/lucene search algorithm with multiple keywords explain this non
> consistent behavior? I would think that adding more keywords would narrow
> down my results.
> 
> I'm pasting below the relevant log in case it helps:
> 
> INFO: [] webapp=/solr path=/select/
> params={spellcheck=true&facet=true&facet.mincount=1&facet.limit=20&spellche
> ck.q=keyword1+keyword2+keyword3+keyword4&
>  json.nl=map&wt=json&version=1.2&rows=10&fl=id,nid,title,comment_count,type
> ,created,changed,score,path,url,uid,name&start=0&facet.sort=true&q=keyword1
> +keyword2+keyword3+keyword4&bf=recip(rord(created),4,10704,10704)^200.0&fac
> et.field=im_cck_field_author&facet.field=type&facet.field=im_vid_1=&indent=
> on&start=0&version=2.2&rows=10} hits=10704 status=0 QTime=1
> 
> any hint on whether this is expected or not appreciated.
> 
> Steph.
> 

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Garbled facets even in a zero hit search

[Dennis Schafroth]

I am definitely not excluding the idea that index is garbled, but.. it doesn't 
explain that I get facets on zero hit.

The schema is as following: 


satay.xml
Description: XML document


where I have copied field for the facets (author_exact, subject_exact, 
title_exact), as I don't want tokenization on these. 

the request has the following parameters: 
facet=true&start=0&q=title:xyzy&f.date.facet.limit=10&f.subject_exact.facet.limit=10&facet.field=author_exact&facet.field=date&facet.field=subject_exact&f.author_exact.facet.limit=10&rows=20

I haven't been able to reproduce it in a test index yet, but I do have two 
different index that show similar problem (facets on zero hits).  

cheers, 
:-Dennis Schafroth

On 09/09/2010, at 15.10, Erick Erickson wrote:

> That looks...er...unfortunate. The very first thing I'd do is check your
> index
> and see if there are such weird values in your facet fields. My guess is
> that SOLR is working fine, but you somehow have garbage values
> in your index, but that's only a guess. I'd try that before trying to
> debug, GIGO.
> 
> Which wouldn't answer the question of how the garbage got in there
> in the first place, posting your field type definition for your
> faceted fields would help with that question.
> 
> Best
> Erick
> 
> On Thu, Sep 9, 2010 at 8:19 AM, Dennis Schafroth wrote:
> 
>> Hi,
>> 
>> Running on a Debian 5.0.5 64bit box. Using
>> solr-1.4.1 with Java version "1.6.0_20"
>> 
>> I am seeing weird facets results along with the "right" looking ones.
>> Garbled data, stuff that looks like a buffer overflow / index off by ...
>> 
>> And I even get them when I do a zero hit search. I wouldn't expect any
>> facets:
>> 
>> 
>> 
>>  
>>0
>>56
>>
>>  true
>>  satay:8985/solr
>>  0
>>  title:xzyzx
>>  10
>>  10
>>  
>>author_exact
>>date
>>subject_exact
>>  
>>  10
>>  20
>>
>>  
>>  
>>  
>>
>>
>>  
>>0
>>0
>>0> int>
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>  
>>  
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>  
>>  
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>0
>>  
>>
>>
>>  
>> 
>> 
>> 
>> I tried to look for a bug report, but haven't been able to find one that 
>> matches. I will try to setup a debug session to get closer, but would love 
>> to get feedback if this is a know issue.
>> 
>> cheers,
>> 
>> :-Dennis Schafroth
>> 
>> 
>> 

AW: SolrException log

[Bastian]

Hi Tommaso, hi solr-users,

i raised both maxCommitAge and maxCommitsToKeep and tracked the occurance of
the "read past EOF Exception". i started with
2 commits and 60MINUTES, and now im at 15commits and 180MINUTES, but with no
luck, the Exception still pops up in nearly 
same frequency as before (approx every 2-3 days).

Any other ideas i should give a try?

-Ursprüngliche Nachricht-
Von: Tommaso Teofili [mailto:tommaso.teof...@gmail.com] 
Gesendet: Mittwoch, 25. August 2010 11:30
An: solr-user@lucene.apache.org
Betreff: Re: SolrException log

Hi again Bastian,

I would advice to arise commit points to a reasonable value considering
indexing (and commit requests) and searching frequencies. Infact keeping too
many commit points would waste disk space but having "enough" should prevent
you from your issue.
I would do some tests with small values of maxCommitsToKeep (no more than
10/20) and maxCommitAge to one of the proposed values (30MINUTES or 1DAY)
and see what happens.

Re: Garbled facets even in a zero hit search

[Markus Jelsma]

That's normal behavior if you haven't configured facet.mincount. Check the 
wiki.

On Thursday 09 September 2010 16:05:01 Dennis Schafroth wrote:
> I am definitely not excluding the idea that index is garbled, but.. it
>  doesn't explain that I get facets on zero hit.
> 
> The schema is as following:
> 

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Date faceting +1MONTH problem

[Liam O'Boyle]

Evening,

I'm trying to break down the data over a year into facets by month; to avoid
overlap, I'm using -1MILLI on the start and end dates and using a gap of
+1MONTH.

However, it seems like February completely breaks my monthly cycles, leading
to incorrect counts further down the line; facets that are after February
only go to the 28th of the month, and items in the other two or three days
get pushed into the next facet.  What's the correct way to do this?

An example is shown below, the facet periods go 2008-12-31, 2009-01-31,
2009-02-28 and then from then on only hit 28.

[2008-12-31T23:59:59.999Z] => 0
[2009-01-31T23:59:59.999Z] => 0
[2009-02-28T23:59:59.999Z] => 0
[2009-03-28T23:59:59.999Z] => 0
[2009-04-28T23:59:59.999Z] => 0
[2009-05-28T23:59:59.999Z] => 0
[2009-06-28T23:59:59.999Z] => 0
[2009-07-28T23:59:59.999Z] => 0
[2009-08-28T23:59:59.999Z] => 13
[2009-09-28T23:59:59.999Z] => 6
[2009-10-28T23:59:59.999Z] => 2
[2009-11-28T23:59:59.999Z] => 7
[gap] => +1MONTH
[end] => 2009-12-28T23:59:59.999Z

Thanks for your help,

Liam

Re: how to normalize a query

[Markus Jelsma]

LuceneQParser
http://lucene.apache.org/java/2_4_0/queryparsersyntax.html#Proximity%20Searches

DismaxQParser
http://wiki.apache.org/solr/DisMaxQParserPlugin#qs_.28Query_Phrase_Slop.29


On Thursday 09 September 2010 15:08:41 dan sutton wrote:
> Hi,
> 
> Does anyone know how I might normalized a query so that e.g. q="one two"
> equals q="two one"
> 
> Cheers,
> Dan
> 

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: how to normalize a query

[dan sutton]

What I wanted was a was to determine that simply the query q="one two" is
equivalent to q="two one", by normalizing I might have

q="one two" for both for example, and then the q.hashCode() would be the
same

Simply using q.hashCode()  returns different values for each query above so
this is not suitable

Cheers
Dan

On Thu, Sep 9, 2010 at 3:36 PM, Markus Jelsma wrote:

> LuceneQParser
>
> http://lucene.apache.org/java/2_4_0/queryparsersyntax.html#Proximity%20Searches
>
> DismaxQParser
> http://wiki.apache.org/solr/DisMaxQParserPlugin#qs_.28Query_Phrase_Slop.29
>
>
> On Thursday 09 September 2010 15:08:41 dan sutton wrote:
> > Hi,
> >
> > Does anyone know how I might normalized a query so that e.g. q="one two"
> > equals q="two one"
> >
> > Cheers,
> > Dan
> >
>
> Markus Jelsma - Technisch Architect - Buyways BV
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
>
>

Re: Garbled facets even in a zero hit search

[Dennis Schafroth]

Thanks, that did it. 

On 09/09/2010, at 16.14, Markus Jelsma wrote:

> That's normal behavior if you haven't configured facet.mincount. Check the 
> wiki.
> 
> On Thursday 09 September 2010 16:05:01 Dennis Schafroth wrote:
>> I am definitely not excluding the idea that index is garbled, but.. it
>> doesn't explain that I get facets on zero hit.
>> 
>> The schema is as following:
>> 
> 
> Markus Jelsma - Technisch Architect - Buyways BV
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
> 
> 

Re: how to normalize a query

[Jonathan Rochkind]

Those two queries might NOT always be 'the same', depending on how you 
have your Solr request handler set up.


For instance, if you have dismax with a ps boost, then "two one" may end 
up with different relevancy scores than "one two", because the query as 
a phrase will be used for boosting, and those are two different 
phrases.  That isn't the only example.


In general, I think this isn't a Solr issue, it depends on the semantics 
of your app and setup, and you've just got to write code in the language 
of your choice to determine if two q strings are 'the same' for your 
purposes.


Jonathan

dan sutton wrote:

What I wanted was a was to determine that simply the query q="one two" is
equivalent to q="two one", by normalizing I might have

q="one two" for both for example, and then the q.hashCode() would be the
same

Simply using q.hashCode()  returns different values for each query above so
this is not suitable

Cheers
Dan

On Thu, Sep 9, 2010 at 3:36 PM, Markus Jelsma wrote:

  

LuceneQParser

http://lucene.apache.org/java/2_4_0/queryparsersyntax.html#Proximity%20Searches

DismaxQParser
http://wiki.apache.org/solr/DisMaxQParserPlugin#qs_.28Query_Phrase_Slop.29


On Thursday 09 September 2010 15:08:41 dan sutton wrote:


Hi,

Does anyone know how I might normalized a query so that e.g. q="one two"
equals q="two one"

Cheers,
Dan

  

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350





  

Re: Solr, c/s type ?

[Yonik Seeley]

On Thu, Sep 9, 2010 at 1:20 AM, Jonathan Rochkind  wrote:
> You _could_ use SolrJ with EmbeddedSolrServer.  But personally I wouldn't 
> unless there's a reason to.  There's no automatic reason not to use the 
> ordinary Solr HTTP api, even for an in-house application which is not a web 
> application.  Unless you have a real reason to use embedded solr, I'd use the 
> HTTP api, possibly via SolrJ if your local application is Java.


Right.  Some people associate HTTP with "slow", but it's just not so.
HTTP is just a TCP socket with some small standard headers.
Solr can even talk binary over HTTP so there isn't even text or XML
parsing overhead.

Think of it like a database - the norm is to run it as a separate
server process.
Embedding is only something you should do if you have really
"interesting" requirements.

-Yonik
http://lucenerevolution.org  Lucene/Solr Conference, Boston Oct 7-8

Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

[Grant Ingersoll]

On Sep 6, 2010, at 10:41 AM, Yonik Seeley wrote:

> 
> For SolrCloud, I don't think we'll end up using consistent hashing -
> we don't need it (although some of the concepts may still be useful).

Can you elaborate on why we don't need it?

custom sorting / help overriding FieldComparator

[Brad Dewar]

Hi

I'm trying to get my (overly complex and strange) product IDs sorting properly 
in Solr.

Approaches I've tried so far, that I've given up on for various reasons:
--Normalizing/padding the IDs so they naturally sort 
alphabetically/alphanumerically.
--Splitting the ID into multiple Solr fields and sending a longer, multi-field 
"sort" argument in the GET request.
--(both of those approaches do work "most of the time", but aren't quite 
perfect)

However, in another project, I already have a Comparble class 
defined in Java that represents a ProductID and does sort them correctly every 
time.  It's not yet in lucene/solr, though.  So I'm trying to make a FieldType 
plugin for Solr that uses the existing ProductID class/datatype.

I need some help extending the lucene FieldComparator class.  I don't know much 
about the rest of the solr / lucene codebase, so I'm fumbling around a bit, 
especially with the required setNextReader() method.  setNextReader() looks 
like it checks the FieldCache to see if this value is there already, otherwise 
grabs a bunch of documents from the index.  I think I should call some form of 
FieldCache.getCustom() for this, but FieldCache.getCustom() itself accepts a 
comparator as an argument, and is marked as "@deprecated Please implement 
FieldComparatorSource directly, instead" ... but isn't that what I'm doing?

So, I'm just a bit confused.  Any help?  Specifically, any help implementing a 
setNextReader() method in a customComparator?

(solr 1.4.1 / lucene 2.9.3)

Thanks,
Brad

Re: SolrCloud distributed indexing (Re: anyone use hadoop+solr?)

[Yonik Seeley]

On Thu, Sep 9, 2010 at 11:51 AM, Grant Ingersoll  wrote:
> On Sep 6, 2010, at 10:41 AM, Yonik Seeley wrote:
>> For SolrCloud, I don't think we'll end up using consistent hashing -
>> we don't need it (although some of the concepts may still be useful).
>
> Can you elaborate on why we don't need it?

I guess because I can't think of a reason why we would need it - hence
it seems we don't?

Random node placement and virtual nodes would seem to be a
disadvantage for us since we aren't just a key-value store and care
about more than one key at a time.  Larger partitions in conjunction
with user-based/directed partitioning will allow doing things like
querying a specific user's email box (for example) by hitting a single
(or very few) nodes in the complete cluster.

-Yonik
http://lucenerevolution.org  Lucene/Solr Conference, Boston Oct 7-8

Re: How to extend IndexSchema and SchemaField

[Renaud Delbru]

 Hi,

so I suppose there is no solution. Is there a chance that SchemaField 
becomes extensible in the future ? Because, at the moment, all the field 
attributes (indexed, stored, etc.) are hardcoded inside SchemaField. Do 
you think it is worth opening an issue about it ?

--
Renaud Delbru

On 07/09/10 16:13, Renaud Delbru wrote:

 Hi,

I would like to extend the field node in the schema.xml by adding new 
attributes. For example, I would like to be able to write:


And be able to access myattribute directly from IndexSchema and 
SchemaField objects. However, these two classes are final, and also 
not very easy to extend ?

Is there any other solutions ?

thanks,

Re: Inconsistent search results with multiple keywords

[Stéphane Corlosquet]

Hi Erick,


On Thu, Sep 9, 2010 at 9:41 AM, Erick Erickson wrote:

> Could you show us the  definitions for your fields? I suspect
> you're not getting the tokens you expect. This will almost certainly
> be true if the type is "string" rather than "text".
>

I should mention that I use solr via the Drupal apachesolr module which
ships with schema.xml and solrconfig.xml files. Here are the ones I use:

http://drupalcode.org/viewvc/drupal/contributions/modules/apachesolr/schema.xml?revision=1.1.2.1.2.32.2.6&view=markup

http://drupalcode.org/viewvc/drupal/contributions/modules/apachesolr/solrconfig.xml?revision=1.1.2.18.2.6&view=markup


> The solr admin page (especially analysis) will help you a lot here, as
> will adding &debugQuery=on to your query and seeing how the query
> is actually processed. Showing us the results of this will also help.
>

The results below are from my local solr endpoint with the debugQuery option
(e.g.
http://localhost:8983/solr/select/?q=Synuclein&version=2.2&start=0&rows=10&indent=on&debugQuery=on
)

## 1 keyword, numFound="827"

http://localhost:8983/solr/select/?q=Synuclein&version=2.2&start=0&rows=10&indent=on&debugQuery=on

 Synuclein
 Synuclein
 +DisjunctionMaxQuery((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01)
DisjunctionMaxQuery((body:synuclein^2.0)~0.01)
 +(tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01 (body:synuclein^2.0)~0.01

## 2 keywords, numFound="88"

http://localhost:8983/solr/select/?q=Synuclein+animal&version=2.2&start=0&rows=10&indent=on&debugQuery=on

 Synuclein animal
 Synuclein animal
 +((DisjunctionMaxQuery((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01) DisjunctionMaxQuery((tags_h1:anim^5.0 |
body:anim^40.0 | title:anim^5.0 | tags_h4_h5_h6:anim^2.0 | tags_inline:anim
| name:anim^3.0 | taxonomy_names:anim^2.0 | tags_h2_h3:anim^3.0)~0.01))~2)
DisjunctionMaxQuery((body:"synuclein anim"~15^2.0)~0.01)
 +(((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01 (tags_h1:anim^5.0 | body:anim^40.0 |
title:anim^5.0 | tags_h4_h5_h6:anim^2.0 | tags_inline:anim | name:anim^3.0 |
taxonomy_names:anim^2.0 | tags_h2_h3:anim^3.0)~0.01)~2) (body:"synuclein
anim"~15^2.0)~0.01

## 3 keywords, numFound goes up: numFound="265"

http://localhost:8983/solr/select/?q=Synuclein+animal+dopamine&version=2.2&start=0&rows=10&indent=on&debugQuery=on

 Synuclein animal dopamine
 Synuclein animal dopamine
 +((DisjunctionMaxQuery((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01) DisjunctionMaxQuery((tags_h1:anim^5.0 |
body:anim^40.0 | title:anim^5.0 | tags_h4_h5_h6:anim^2.0 | tags_inline:anim
| name:anim^3.0 | taxonomy_names:anim^2.0 | tags_h2_h3:anim^3.0)~0.01)
DisjunctionMaxQuery((tags_h1:dopamin^5.0 | body:dopamin^40.0 |
title:dopamin^5.0 | tags_h4_h5_h6:dopamin^2.0 | tags_inline:dopamin |
name:dopamin^3.0 | taxonomy_names:dopamin^2.0 |
tags_h2_h3:dopamin^3.0)~0.01))~2) DisjunctionMaxQuery((body:"synuclein anim
dopamin"~15^2.0)~0.01)
 +(((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01 (tags_h1:anim^5.0 | body:anim^40.0 |
title:anim^5.0 | tags_h4_h5_h6:anim^2.0 | tags_inline:anim | name:anim^3.0 |
taxonomy_names:anim^2.0 | tags_h2_h3:anim^3.0)~0.01 (tags_h1:dopamin^5.0 |
body:dopamin^40.0 | title:dopamin^5.0 | tags_h4_h5_h6:dopamin^2.0 |
tags_inline:dopamin | name:dopamin^3.0 | taxonomy_names:dopamin^2.0 |
tags_h2_h3:dopamin^3.0)~0.01)~2) (body:"synuclein anim
dopamin"~15^2.0)~0.01

## 4 keywords, numFound="45"

http://localhost:8983/solr/select/?q=Synuclein+animal+dopamine+calcium&version=2.2&start=0&rows=10&indent=on&debugQuery=on

 Synuclein animal dopamine calcium
 Synuclein animal dopamine calcium
 +((DisjunctionMaxQuery((tags_h1:synuclein^5.0 |
body:synuclein^40.0 | title:synuclein^5.0 | tags_h4_h5_h6:synuclein^2.0 |
tags_inline:synuclein | name:synuclein^3.0 | taxonomy_names:synuclein^2.0 |
tags_h2_h3:synuclein^3.0)~0.01) DisjunctionMaxQuery((tags_h1:anim^5.0 |
body:anim^40.0 | title:anim^5.0 | tags_h4_h5_h6:anim^2.0 | tags_inline:anim
| name:anim^3.0 | taxonomy_names:anim^2.0 | tags_h2_h3:anim^3.0)~0.01)
DisjunctionMaxQuery((tags_h1:dopamin^5.0 | body:

Re: Inconsistent search results with multiple keywords

[Stéphane Corlosquet]

Hi Markus,

On Thu, Sep 9, 2010 at 9:55 AM, Markus Jelsma wrote:

> Looks like AND is your defaultOperator [1].


yes, my schema.xml file have   which
is why I thought that the number of hits would decrease every time you add a
keyword.



> Check your schema.xml and try
> adding q.op=or to your query.
>

Interestingly, whether I add q.op=and, q.op=or or nothing to my query, I
still get the same amount of results.

http://localhost:8983/solr/select/?q=Synuclein+animal+dopamine&version=2.2&start=0&rows=10&indent=on&debugQuery=on&q.op=orreturn
265 results no matter what the value of q.op is.

Steph.



>
> [1]: http://wiki.apache.org/solr/SearchHandler#q.op
>
> On Thursday 09 September 2010 15:34:52 Stéphane Corlosquet wrote:
> > Hi all,
> >
> > I'm new to solr so please let me know if there is a more appropriate
> place
> > for my question below.
> >
> > I'm noticing a rather unexpected number of results when I add more
> keywords
> > to a search. I'm listing below a example (where I replaced the real
> >  keywords with placeholders):
> >
> > keyword1 851 hits
> > keyword1 keyword2  90 hits
> > keyword1 keyword2 keyword3 269 hits
> > keyword1 keyword2 keyword3 keyword4 47 hits
> >
> > As you can see, adding k2 narrows down the amount of results (as I would
> > expect), but adding k3 to k1 and k2 suddenly increases the amount of
> > results. with 4 keywords, the results have been narrowed down again.
> Would
> > solr/lucene search algorithm with multiple keywords explain this non
> > consistent behavior? I would think that adding more keywords would narrow
> > down my results.
> >
> > I'm pasting below the relevant log in case it helps:
> >
> > INFO: [] webapp=/solr path=/select/
> >
> params={spellcheck=true&facet=true&facet.mincount=1&facet.limit=20&spellche
> > ck.q=keyword1+keyword2+keyword3+keyword4&
> >  json.nl
> =map&wt=json&version=1.2&rows=10&fl=id,nid,title,comment_count,type
> >
> ,created,changed,score,path,url,uid,name&start=0&facet.sort=true&q=keyword1
> >
> +keyword2+keyword3+keyword4&bf=recip(rord(created),4,10704,10704)^200.0&fac
> >
> et.field=im_cck_field_author&facet.field=type&facet.field=im_vid_1=&indent=
> > on&start=0&version=2.2&rows=10} hits=10704 status=0 QTime=1
> >
> > any hint on whether this is expected or not appreciated.
> >
> > Steph.
> >
>
> Markus Jelsma - Technisch Architect - Buyways BV
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
>
>

Re: svn branch issues

[Chris Hostetter]

:   $ svn co
: http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev
:   svn: Repository moved permanently to
: '/viewvc/lucene/solr/branches/branch-1.5-dev/'; please relocate

those aren't SVN URLs, those are urls for the "viewvc" SVN Browsing tool

your SVN client is trying to interpret the 301 redirect from 
http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev to 
http://svn.apache.org/viewvc/lucene/solr/branches/branch-1.5-dev/ (which 
your browser does automaticly)

The actual URL of the SVN repository (for the old 1.5 branch) is

http://svn.apache.org/repos/asf/lucene/solr/branches/branch-1.5-dev/

-Hoss

--
http://lucenerevolution.org/  ...  October 7-8, Boston
http://bit.ly/stump-hoss  ...  Stump The Chump!

roadmap

[Lukas Kahwe Smith]

Hi,

With the Lucene svn merge a lot of tentative release dates seemed to have 
slipped. Which is fine, because I think the merge is for the greater good of 
both projects in the long run.

However I do subscribe to the school of thought that believes OSS is best 
served with a release often mantra. Of course such a one time restructure can 
add a few months. So right now the main thing I feel a lot of people are 
wanting to hear is a tenative timeline for when to expect the next release and 
the key features that we can expect.

At least looking at http://lucene.apache.org/solr/ I do not see anything that 
communicates to the users where things are heading. Or am I just looking in the 
wrong place?

I hope I am not coming off as a whiney user, again I am not telling you guys to 
work harder without me handing you a pay check. I am just suggesting that a bit 
more transparency as to whats going to happen in the near future would make it 
all the more easier for us users to bet our futures on solr :)

regards,
Lukas Kahwe Smith
m...@pooteeweet.org

Using more than one name for a query field - aliases

[Shawn Heisey]

 I find myself in need of the ability to access one field by more than 
one name, for application transition purposes.  Right now we have a 
field (ft_text, by far the largest part of the index) that is indexed 
but not stored.  This field and three others are copied into an 
additional field called catchall, also indexed but not stored, which is 
the default search field.


I would like to drop ft_text and make each index shard 3GB smaller, but 
make it so that any queries which use ft_text get automatically 
redirected to catchall.  Ultimately we will be replacing catchall with 
dismax and eliminating it.  After the switch to dismax is complete and 
catchall is gone, I want to switch back to using ft_text for specific 
searches generated by the application.


Is this currently possible?  I saw SOLR-1205, but this seems more 
concerned with query output, not query input.


Thanks,
Shawn

Re: Inconsistent search results with multiple keywords

[Ahmet Arslan]

> yes, my schema.xml file have   defaultOperator="AND"/> which
> is why I thought that the number of hits would decrease
> every time you add a
> keyword.

You are using dismax so, it is determined by mm parameter.

http://wiki.apache.org/solr/DisMaxQParserPlugin#mm_.28Minimum_.27Should.27_Match.29

RE: Re: Inconsistent search results with multiple keywords

[Markus Jelsma]

Indeed, it's the dismax, i missed it! My bad.. 
 
-Original message-
From: Ahmet Arslan 
Sent: Thu 09-09-2010 20:37
To: solr-user@lucene.apache.org; 
Subject: Re: Inconsistent search results with multiple keywords

> yes, my schema.xml file have   defaultOperator="AND"/> which
> is why I thought that the number of hits would decrease
> every time you add a
> keyword.

You are using dismax so, it is determined by mm parameter.

http://wiki.apache.org/solr/DisMaxQParserPlugin#mm_.28Minimum_.27Should.27_Match.29


     

RE: roadmap

[Markus Jelsma]

You should check Jira's roadmap [1] instead. It shows a clear picture of what 
has been done since the 1.4.1 release and pending issues for the 3.x branch and 
others.

 

[1]: 
https://issues.apache.org/jira/browse/SOLR?report=com.atlassian.jira.plugin.system.project:roadmap-panel

 


 
-Original message-
From: Lukas Kahwe Smith 
Sent: Thu 09-09-2010 20:20
To: solr-user@lucene.apache.org; 
Subject: roadmap

Hi,

With the Lucene svn merge a lot of tentative release dates seemed to have 
slipped. Which is fine, because I think the merge is for the greater good of 
both projects in the long run.

However I do subscribe to the school of thought that believes OSS is best 
served with a release often mantra. Of course such a one time restructure can 
add a few months. So right now the main thing I feel a lot of people are 
wanting to hear is a tenative timeline for when to expect the next release and 
the key features that we can expect.

At least looking at http://lucene.apache.org/solr/ I do not see anything that 
communicates to the users where things are heading. Or am I just looking in the 
wrong place?

I hope I am not coming off as a whiney user, again I am not telling you guys to 
work harder without me handing you a pay check. I am just suggesting that a bit 
more transparency as to whats going to happen in the near future would make it 
all the more easier for us users to bet our futures on solr :)

regards,
Lukas Kahwe Smith
m...@pooteeweet.org

Re: roadmap

[Lukas Kahwe Smith]

On 09.09.2010, at 20:47, Markus Jelsma wrote:

> You should check Jira's roadmap [1] instead. It shows a clear picture of what 
> has been done since the 1.4.1 release and pending issues for the 3.x branch 
> and others.
> 
>  
> 
> [1]: 
> https://issues.apache.org/jira/browse/SOLR?report=com.atlassian.jira.plugin.system.project:roadmap-panel


good point. then again this link leaves a lot of open questions. for one i have 
heard many core solr guys say that 1.5 will likely never be released. am i just 
following a rumor here, but I guess the "now defunt" note would imply that I 
guess all the 1.5 tickets will just move to whatever comes next? also it doesnt 
mention tentative release schedules either.

anyway, i think it for now a little news port on the main page with a short 
explanation what happened and what the key goal (new features, improvements) 
are for the next release and a ball park when to expect them would go a long 
way.

regards,
Lukas Kahwe Smith
m...@pooteeweet.org

RE: Re: Re: Invariants on a specific fq value

[Markus Jelsma]

It works as expected. The append, well, appends the parameter and because each 
collection has a unique value, specifying two filters on different collections 
will always yield zero results.

 

This, of course, won't work for values that are shared between collections.
 
-Original message-
From: Yonik Seeley 
Sent: Wed 08-09-2010 19:38
To: solr-user@lucene.apache.org; 
Subject: Re: Re: Invariants on a specific fq value

2010 at 1:32 PM, Markus Jelsma  wrote:
> Interesting! I haven't met the appends method before and i'll be sure to give 
> it a try tomorrow. Try, the wiki [1] is not very clear on what it really does.

Here's a comment from the example solrconfig.xml:

   

-Yonik
http://lucenerevolution.org  Lucene/Solr Conference, Boston Oct 7-8

Re: Inconsistent search results with multiple keywords

[Stéphane Corlosquet]

Thank you Erick, Markus and Ahmet! That answered my question. Changing the
value of the mm parameter in solrconfig.xml did have an effect on the 3
keyword query (it was set to 2->25%), and removing it entirely forced all
keywords to be present, and the number of hits was decreasing as expected.

I'm impressed by the reactivity on this list =) - great job!

Steph.

On Thu, Sep 9, 2010 at 2:36 PM, Ahmet Arslan  wrote:

> > yes, my schema.xml file have   > defaultOperator="AND"/> which
> > is why I thought that the number of hits would decrease
> > every time you add a
> > keyword.
>
> You are using dismax so, it is determined by mm parameter.
>
>
> http://wiki.apache.org/solr/DisMaxQParserPlugin#mm_.28Minimum_.27Should.27_Match.29
>
>
>
>

Re: Inconsistent search results with multiple keywords

[Stéphane Corlosquet]

sorry, mm was set to 2<-35%, not 2->25%, but nevermind.

Steph.

On Thu, Sep 9, 2010 at 3:13 PM, Stéphane Corlosquet
wrote:

> Thank you Erick, Markus and Ahmet! That answered my question. Changing the
> value of the mm parameter in solrconfig.xml did have an effect on the 3
> keyword query (it was set to 2->25%), and removing it entirely forced all
> keywords to be present, and the number of hits was decreasing as expected.
>
> I'm impressed by the reactivity on this list =) - great job!
>
> Steph.
>
> On Thu, Sep 9, 2010 at 2:36 PM, Ahmet Arslan  wrote:
>
>> > yes, my schema.xml file have  > > defaultOperator="AND"/> which
>> > is why I thought that the number of hits would decrease
>> > every time you add a
>> > keyword.
>>
>> You are using dismax so, it is determined by mm parameter.
>>
>>
>> http://wiki.apache.org/solr/DisMaxQParserPlugin#mm_.28Minimum_.27Should.27_Match.29
>>
>>
>>
>>
>

Re: Delta Import with something other than Date

[Shawn Heisey]

 On 9/8/2010 4:32 PM, David Yang wrote:

I have a table that I want to index, and the table has no datetime
stamp. However, the table is append only so the primary key can only go
up. Is it possible to store the last primary key, and use some delta
query="select id where id>${last_id_value}"



I ran into the same thing.  I track this in my build scripts and simply 
pass min and max ID variables via the dataimport URL, which are used in 
data-config.xml to plug them into my SQL statement.  When I asked about 
it on the list, someone important told me to file a Jira on making it 
generic, it is SOLR-1920.


https://issues.apache.org/jira/browse/SOLR-1920

Thanks,
Shawn

RE: Delta Import with something other than Date

[Vladimir Sutskever]

Shawn,

Can you provide a sample of passing the parameter via URL? And how using it 
would look in the data-config.xml

Thanks!

-Vladimir


-Original Message-
From: Shawn Heisey [mailto:elyog...@elyograg.org] 
Sent: Thursday, September 09, 2010 3:04 PM
To: solr-user@lucene.apache.org
Subject: Re: Delta Import with something other than Date

  On 9/8/2010 4:32 PM, David Yang wrote:
> I have a table that I want to index, and the table has no datetime
> stamp. However, the table is append only so the primary key can only go
> up. Is it possible to store the last primary key, and use some delta
> query="select id where id>${last_id_value}"
>

I ran into the same thing.  I track this in my build scripts and simply 
pass min and max ID variables via the dataimport URL, which are used in 
data-config.xml to plug them into my SQL statement.  When I asked about 
it on the list, someone important told me to file a Jira on making it 
generic, it is SOLR-1920.

https://issues.apache.org/jira/browse/SOLR-1920

Thanks,
Shawn

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

Extending org.apache.solr.hander.dataimport.Transformer

[Vladimir Sutskever]

Hi All,

I am building a custom DIH transformer to massage the data before its indexed.

Extending org.apache.solr.hander.dataimport.Transformer requires me to 
implement the

transformRow(Map row, Context context)


During the actual import  - SOLR complains because its looking for method with 
signature transformRow(Map row)


Any idea why it looks for that method signature instead of the 
transformRow(Map row, Context context) ?





Kind regards,

Vladimir Sutskever
Investment Bank - Technology
JPMorgan Chase, Inc.



This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  

Help on spelling.

[Gregg Hoshovsky]

I am trying to use the spellchecker but cannot get past the point of having the 
spelling possibilities returned.

I have a text field define in the schema.xml file as:

   

I modified solrconfig.xml to point the analyzer to the same field type and have 
the name set the same.

  

text_ws


  default
  text
  ./spellchecker



I left the handler alone

  


I see that the spellchecker folder gets files built so I am assuming that the 
spelling data is being created

Then I ran the query as
http://localhost:8983/solr/biolibrary/spell/?q=text:wedg&version=2.2&start=0&rows=10&indent=on&wt=json

I would expect that this would have returned some spelling suggestions ( such 
as wedge) but don’t get anything besides:

{
 "responseHeader":{
  "status":0,
  "QTime":1},
 "response":{"numFound":0,"start":0,"docs":[]
 }}

Any help is appreciated.

Gregg

Configurable dataimport.properties location in Solr 1.4

[Manali Joshi]

Hi,

I am using solr 1.4 and was wondering if the dataimport.properties location
can be configured for DataImportHandler. Is there a config file or a
property that I can use to specify a custom location ?

Thanks,
Manali

RE: Help on spelling.

[Markus Jelsma]

I don't see you passing spellcheck parameters in the query string. Are they 
configured as default in your search handler?
 
-Original message-
From: Gregg Hoshovsky 
Sent: Thu 09-09-2010 22:40
To: solr-user@lucene.apache.org; 
Subject: Help on spelling.

I am trying to use the spellchecker but cannot get past the point of having the 
spelling possibilities returned.

I have a text field define in the schema.xml file as:

  

I modified solrconfig.xml to point the analyzer to the same field type and have 
the name set the same.

 

   text_ws

   
     default
     text
     ./spellchecker
   


I left the handler alone

 
   

I see that the spellchecker folder gets files built so I am assuming that the 
spelling data is being created

Then I ran the query as
http://localhost:8983/solr/biolibrary/spell/?q=text:wedg&version=2.2&start=0&rows=10&indent=on&wt=json

I would expect that this would have returned some spelling suggestions ( such 
as wedge) but don t get anything besides:

{
"responseHeader":{
 "status":0,
 "QTime":1},
"response":{"numFound":0,"start":0,"docs":[]
}}

Any help is appreciated.

Gregg

Re: Is there a way to fetch the complete list of data from a particular column in SOLR document?

[Scott K]

But how do you know when the document actually makes it to solr,
especially if you are using commitWithin and not explicitly calling
commit.

One solution is to have a status field in the database such as
0 - unindexed
1 - indexing
2 - committed / verified

And have a separate process query solr for documents in the indexing
state and set them to committed if they are queryable in solr.

On Tue, Sep 7, 2010 at 14:26, Geert-Jan Brits  wrote:
>>Please let me know if there are any other ideas / suggestions to implement
> this.
>
> You're indexing program should really take care of this IMHO. Each time your
> indexer inserts a document to Solr, flag the corresponding entity in your
> RDBMS, each time you delete, remove the flag. You should implement this as a
> transaction to make sure all is still fine in the unlikely event of a crash
> midway.
>
> 2010/9/7 bbarani 
>
>>
>> Hi,
>>
>> I am trying to get complete list of unique document ID and compare it with
>> that of back end to make sure that both back end and SOLR documents are in
>> sync.
>>
>> Is there a way to fetch the complete list of data from a particular column
>> in SOLR document?
>>
>> Once I get the list, I can easily compare it against the DB and delete the
>> orphan documents..
>>
>> Please let me know if there are any other ideas / suggestions to implement
>> this.
>>
>> Thanks,
>> Barani
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Is-there-a-way-to-fetch-the-complete-list-of-data-from-a-particular-column-in-SOLR-document-tp1435586p1435586.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>

How to load data to embedded server

[Rico Lelina]

Hi,

I'm using EmbeddedSolrServer for my unit tests. I just can't figure out how to 
add my data (stored in xml files similar to those in the example application 
example/exampleDocs) after instantiating the server. The source code for the 
simple post tool seems to require a stream to write the file contents to. How 
can I do that with the embedded server?

Thanks,
Rico

Building query based on value of boolean field

[PeterKerk]

I have this field in my schema.xml:


This one in my data-config:
   

Now, how can I return all results for which partylocation = true?

Thanks!
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Building-query-based-on-value-of-boolean-field-tp1449018p1449018.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Is there a way to fetch the complete list of data from a particular column in SOLR document?

[Geert-Jan Brits]

You're right for the general case. I should have added that our setup is
perhaps a little bit out of the ordinary in that we send explicit commits to
solr as part of our indexing app.
Once a commit has finished we're sure all docs until then are present in
solr. For us it's much more difficult to do the way you suggested bc we
index into several embedded solr shards, etc. It can be done it's just not
convienient. But for the general case I admit querying all ids as a
post-process is probably the more elegant and robust way.

2010/9/9 Scott K 

> But how do you know when the document actually makes it to solr,
> especially if you are using commitWithin and not explicitly calling
> commit.
>
> One solution is to have a status field in the database such as
> 0 - unindexed
> 1 - indexing
> 2 - committed / verified
>
> And have a separate process query solr for documents in the indexing
> state and set them to committed if they are queryable in solr.
>
> On Tue, Sep 7, 2010 at 14:26, Geert-Jan Brits  wrote:
> >>Please let me know if there are any other ideas / suggestions to
> implement
> > this.
> >
> > You're indexing program should really take care of this IMHO. Each time
> your
> > indexer inserts a document to Solr, flag the corresponding entity in your
> > RDBMS, each time you delete, remove the flag. You should implement this
> as a
> > transaction to make sure all is still fine in the unlikely event of a
> crash
> > midway.
> >
> > 2010/9/7 bbarani 
> >
> >>
> >> Hi,
> >>
> >> I am trying to get complete list of unique document ID and compare it
> with
> >> that of back end to make sure that both back end and SOLR documents are
> in
> >> sync.
> >>
> >> Is there a way to fetch the complete list of data from a particular
> column
> >> in SOLR document?
> >>
> >> Once I get the list, I can easily compare it against the DB and delete
> the
> >> orphan documents..
> >>
> >> Please let me know if there are any other ideas / suggestions to
> implement
> >> this.
> >>
> >> Thanks,
> >> Barani
> >> --
> >> View this message in context:
> >>
> http://lucene.472066.n3.nabble.com/Is-there-a-way-to-fetch-the-complete-list-of-data-from-a-particular-column-in-SOLR-document-tp1435586p1435586.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >
>

Re: Date faceting +1MONTH problem

[Chris Hostetter]

: I'm trying to break down the data over a year into facets by month; to avoid
: overlap, I'm using -1MILLI on the start and end dates and using a gap of
: +1MONTH.
: 
: However, it seems like February completely breaks my monthly cycles, leading

Yep.

Everything you posted makes sense to me in how DateMath works - "Jan 31 @ 
23:59.999" + "1 MONTH" results in "Feb 28 @ 23:59.999" ... at which point 
adding "1 MONTH" to that results in "Mar 28 @ ..." because there is no 
context of what the initial starting point was.

It's not a situation i've ever personally run into ... one workarround 
would be to use a "+1MILLI" fudge factor at indexing time, instead of a 
"-1MILLI" fudge factor at query time ... that shouldn't have this problem.

If you'd like to open a bug to trak this, I think it might be possible to 
fix this behavior (there are some features in the Java calendaring code 
that make things like "Jan 31 + 2 Months" do the right thing) but 
personally I think working on SOLR-1896 (combined with the new 
facet.range.include param) is a more effective use of time so 
we can eliminate the need for this type of hack completely in future Solr 
releases.

-Hoss

--
http://lucenerevolution.org/  ...  October 7-8, Boston
http://bit.ly/stump-hoss  ...  Stump The Chump!

PatternReplaceCharFilterFactory?

[Andrew Cogan]

I didn't build SOLR, I downloaded a prebuilt zip file. Am I correct in my
understanding that the charFilter class PatternReplaceCharFilterFactory is
not part of the prebuilt SOLR 1.4.1 distribution? If that's right, how do I
go about adding it? The more explicit the instructions the better, as I have
no prior Java experience.

 

Thank you,

Andrew Cogan

 

Re: Help on spelling.

[Gregg Hoshovsky]

Okay putting "spellcheck=true" makes all the difference in the world.

 Thanks


On 9/9/10 1:58 PM, "Markus Jelsma"  wrote:

> I don't see you passing spellcheck parameters in the query string. Are they
> configured as default in your search handler?
>  
> -Original message-
> From: Gregg Hoshovsky 
> Sent: Thu 09-09-2010 22:40
> To: solr-user@lucene.apache.org;
> Subject: Help on spelling.
> 
> I am trying to use the spellchecker but cannot get past the point of having
> the spelling possibilities returned.
> 
> I have a text field define in the schema.xml file as:
> 
>    multiValued="true"/>
> 
> I modified solrconfig.xml to point the analyzer to the same field type and
> have the name set the same.
> 
>  
> 
>    text_ws
> 
>    
>      default
>      text
>      ./spellchecker
>    
> 
> 
> I left the handler alone
> 
>  
>    
> 
> I see that the spellchecker folder gets files built so I am assuming that the
> spelling data is being created
> 
> Then I ran the query as
> http://localhost:8983/solr/biolibrary/spell/?q=text:wedg&version=2.2&start=0&r
> ows=10&indent=on&wt=json
> 
> I would expect that this would have returned some spelling suggestions ( such
> as wedge) but don t get anything besides:
> 
> {
> "responseHeader":{
>  "status":0,
>  "QTime":1},
> "response":{"numFound":0,"start":0,"docs":[]
> }}
> 
> Any help is appreciated.
> 
> Gregg
> 

Re: PatternReplaceCharFilterFactory?

[Erick Erickson]

Could you give us an idea of why you think it isn't present? As far as I can
tell,
it's been around for a while. Are you getting an error and if so, can you
show it
to us?

Look in schema.xml of what you downloaded (probably in the example
directory).
Is it mentioned there? If so, it should "just be there" but do note that I'm
looking at
the latest trunk version.

Best
Erick

On Thu, Sep 9, 2010 at 6:55 PM, Andrew Cogan wrote:

> I didn't build SOLR, I downloaded a prebuilt zip file. Am I correct in my
> understanding that the charFilter class PatternReplaceCharFilterFactory is
> not part of the prebuilt SOLR 1.4.1 distribution? If that's right, how do I
> go about adding it? The more explicit the instructions the better, as I
> have
> no prior Java experience.
>
>
>
> Thank you,
>
> Andrew Cogan
>
>
>
>

Re: Date faceting +1MONTH problem

[Liam O'Boyle]

Hi Chris,

Yes, I saw the facet.range.include feature and briefly tried to implement it
before realising that it was Solr 3.1 only :)  I agree that it seems like
the best solution to problem.

Reindexing with a +1MILLI hack had occurred to me and I guess that's what
I'll do in the meantime; it just seemed like something that people must have
run into before!  I suppose it depends on the granularity of your
timestamps; all of my values are actually just dates, so I've been putting
them in as the date with T00:00:00.000Z, which makes the overlap problem
very obvious.

If anyone else has come across a solution for this, feel free to suggest
another approach, otherwise it's reindexing time.

Cheers,
Liam


On Fri, Sep 10, 2010 at 8:38 AM, Chris Hostetter
wrote:

> : I'm trying to break down the data over a year into facets by month; to
> avoid
> : overlap, I'm using -1MILLI on the start and end dates and using a gap of
> : +1MONTH.
> :
> : However, it seems like February completely breaks my monthly cycles,
> leading
>
> Yep.
>
> Everything you posted makes sense to me in how DateMath works - "Jan 31 @
> 23:59.999" + "1 MONTH" results in "Feb 28 @ 23:59.999" ... at which point
> adding "1 MONTH" to that results in "Mar 28 @ ..." because there is no
> context of what the initial starting point was.
>
> It's not a situation i've ever personally run into ... one workarround
> would be to use a "+1MILLI" fudge factor at indexing time, instead of a
> "-1MILLI" fudge factor at query time ... that shouldn't have this problem.
>
> If you'd like to open a bug to trak this, I think it might be possible to
> fix this behavior (there are some features in the Java calendaring code
> that make things like "Jan 31 + 2 Months" do the right thing) but
> personally I think working on SOLR-1896 (combined with the new
> facet.range.include param) is a more effective use of time so
> we can eliminate the need for this type of hack completely in future Solr
> releases.
>
> -Hoss
>
> --
> http://lucenerevolution.org/  ...  October 7-8, Boston
> http://bit.ly/stump-hoss  ...  Stump The Chump!
>
>

Re: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : SingleInstanceLock: write.lock

[Lance Norskog]

Hello-

There were a few bugs in this area that are fixed in Solr 1.4. There are 
many other bugs which were also fixed. We suggest everyone upgrade to 1.4.


There are different locking managers, and you may be able to use a 
different one. Also, if this is over NFS that can cause further problems.


Lance

Bharat Jain wrote:

Hi,

We are using SOLR 1.3 and getting this error often. There is only one
instance that index the data. I did some analysis which I have put below and
scenario when this error can happen. Can you guys please validate the issue?
thanks a lot in advance.

SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out
: SingleInstanceLock: write.lock
 at org.apache.lucene.store.Lock.obtain(Lock.java:85)
 at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140)
 at org.apache.lucene.index.IndexWriter.(IndexWriter.java:938)
 at
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:116)
 at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHand
ler.java:122)
 at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHa
ndler2.java:167)
 at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
r2.java:221)

I think this error can happen in the following scneario

1. Thread T1 enters the commit method and its actually an optimize request.
 a. T1 gets the iwCommit lock
 b. T1 enters the openWriter() method
 c. T1 does the (writer == null check) ->  time x1

2. Thread T2 enters the addDoc method
 a. T2 gets iwAccess lock
 b. T2 gets the mutex "this"
 c. T2 enters operWriter method
 d. T2 does the (writer == null check) ->  time x1

Now after the 1.c is done thread yields and 2.d gets execution and it also
sees writer as null and so now both threads will try to create the
indexwriter and it will fail. I have pasted the relevant portion of code
here.


   // iwCommit protects internal data and open/close of the IndexWriter and
   // is a mutex. Any use of the index writer should be protected by
iwAccess,
   // which admits multiple simultaneous acquisitions.  iwAccess is
   // mutually-exclusive with the iwCommit lock.
   protected final Lock iwAccess, iwCommit;



// must only be called when iwCommit lock held
   protected void openWriter() throws IOException {
 if (writer==null) {
   writer = createMainIndexWriter("DirectUpdateHandler2", false);
 }
   }

addDoc(...) {
...

iwAccess.lock();
 try {

   // We can't use iwCommit to protect internal data here, since it would
   // block other addDoc calls.  Hence, we synchronize to protect
internal
   // state.  This is safe as all other state-changing operations are
   // protected with iwCommit (which iwAccess excludes from this block).
   synchronized (this) {
 // adding document -- prep writer
 closeSearcher();
 openWriter();
 tracker.addedDocument();
   } // end synchronized block

   // this is the only unsynchronized code in the iwAccess block, which
   // should account for most of the time

}

commit() {
...

iwCommit.lock();
 try {
   log.info("start "+cmd);

   if (cmd.optimize) {
 closeSearcher();
 openWriter();
 writer.optimize(cmd.maxOptimizeSegments);
   }

   closeSearcher();
   closeWriter();

   callPostCommitCallbacks();
}

Thanks
Bharat Jain

   

Re: How to load data to embedded server

[Lance Norskog]

The stream.file and stream.url parameters should do this.

Lance

Rico Lelina wrote:

Hi,

I'm using EmbeddedSolrServer for my unit tests. I just can't figure out how to
add my data (stored in xml files similar to those in the example application
example/exampleDocs) after instantiating the server. The source code for the
simple post tool seems to require a stream to write the file contents to. How
can I do that with the embedded server?

Thanks,
Rico

   

Re: Delta Import with something other than Date

[Shawn Heisey]

 On 9/9/2010 1:23 PM, Vladimir Sutskever wrote:

Shawn,

Can you provide a sample of passing the parameter via URL? And how using it 
would look in the data-config.xml



Here's the URL that I send to do a full build on my last shard:

http://idxst5-a:8983/solr/build/dataimport?command=full-import&optimize=true&commit=true&dataTable=ncdat&numShards=6&modVal=5&minDid=0&maxDid=242895591

If I want to do a delta, I just change the command to delta-import and 
give it a proper minDid value, rather than 0.


Below is the entity from my data-config.xml.  You have to have a 
deltaQuery defined for delta-import to work, but if you're going to use 
your own placeholders, just put something in that returns a single value 
very quickly.  In my case, my query and deltaImportQuery are actually 
identical.


  query="SELECT *,FROM_UNIXTIME(post_date) as pd FROM 
${dataimporter.request.dataTable} WHERE did > 
${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})"

  deltaQuery="SELECT MAX(did) FROM ${dataimporter.request.dataTable}"
  deltaImportQuery="SELECT *,FROM_UNIXTIME(post_date) as pd FROM 
${dataimporter.request.dataTable} WHERE did > 
${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})">

Re: PatternReplaceCharFilterFactory?

[Shawn Heisey]

 On 9/9/2010 5:38 PM, Erick Erickson wrote:

Could you give us an idea of why you think it isn't present? As far as I can
tell,
it's been around for a while. Are you getting an error and if so, can you
show it
to us?

Look in schema.xml of what you downloaded (probably in the example
directory).
Is it mentioned there? If so, it should "just be there" but do note that I'm
looking at
the latest trunk version.

Best
Erick

On Thu, Sep 9, 2010 at 6:55 PM, Andrew Coganwrote:


I didn't build SOLR, I downloaded a prebuilt zip file. Am I correct in my
understanding that the charFilter class PatternReplaceCharFilterFactory is
not part of the prebuilt SOLR 1.4.1 distribution? If that's right, how do I
go about adding it? The more explicit the instructions the better, as I
have
no prior Java experience.



The one called PatternReplaceFilterFactory (no Char) has been around 
forever.  It is not mentioned on the Wiki page about analyzers.  The one 
called PatternReplaceCharFilterFactory is only available from svn.


I found that the CharFilter version paid no attention to the place in my 
analyzer where it was defined, it went first, even before the tokenizer. 
Furthermore, it didn't actually work, even if I used a simple search and 
replace, it completely ate my source string and nothing went into the 
index.  Here's what I ended up going with, which works with a completely 
unmodified 1.4.1 warfile:




Thanks,
Shawn

Change what gets logged when service is disabled

[Shawn Heisey]

 I use the PingRequestHandler option that tells my load balancer 
whether a machine is available.


When the service is disabled, every one of those requests, which my load 
balancer makes every five seconds, results in the following in the log:


Sep 9, 2010 6:06:58 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Service disabled
at 
org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:48)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)

at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at 
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)


This seems highly excessive, especially for something that I did on 
purpose.  I run with logging at WARN.  Would it make sense to change 
this to an INFO or DEBUG and eliminate the stack trace?  I have minimal 
Java skills, but I am passable with other languages.  Would it be hard 
to find and fix this in the code, given that limitation?


Ultimately I think the severity of this log message should be 
configurable.  I do not think the stack trace is at all helpful, but I 
suppose some people might want a knob to turn that on and off as well.


Thanks,
Shawn

Re: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : SingleInstanceLock: write.lock

[Mark Miller]

iwAccess is the reader lock to iwCommit's writer lock - so the scenario
you bring up should be protected - the reader lock is used in only one
place in the class (addDoc), while every other call to openWriter is
protected by the writer lock.

I'd worry more about the case where two add documents hit at the same
time when the indexWriter is null - then, because both calls to
openWriter are protected with the reader lock, you could race - but
that's what the synchronized(this) protects against ;)

It all looks good to me.

- Mark

On 9/9/10 4:17 AM, Bharat Jain wrote:
> Hi,
> 
> We are using SOLR 1.3 and getting this error often. There is only one
> instance that index the data. I did some analysis which I have put below and
> scenario when this error can happen. Can you guys please validate the issue?
> thanks a lot in advance.
> 
> SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
> out
> : SingleInstanceLock: write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:85)
> at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:938)
> at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:116)
> at
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHand
> ler.java:122)
> at
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHa
> ndler2.java:167)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
> r2.java:221)
> 
> I think this error can happen in the following scneario
> 
> 1. Thread T1 enters the commit method and its actually an optimize request.
> a. T1 gets the iwCommit lock
> b. T1 enters the openWriter() method
> c. T1 does the (writer == null check) -> time x1
> 
> 2. Thread T2 enters the addDoc method
> a. T2 gets iwAccess lock
> b. T2 gets the mutex "this"
> c. T2 enters operWriter method
> d. T2 does the (writer == null check) -> time x1
> 
> Now after the 1.c is done thread yields and 2.d gets execution and it also
> sees writer as null and so now both threads will try to create the
> indexwriter and it will fail. I have pasted the relevant portion of code
> here.
> 
> 
>   // iwCommit protects internal data and open/close of the IndexWriter and
>   // is a mutex. Any use of the index writer should be protected by
> iwAccess,
>   // which admits multiple simultaneous acquisitions.  iwAccess is
>   // mutually-exclusive with the iwCommit lock.
>   protected final Lock iwAccess, iwCommit;
> 
> 
> 
> // must only be called when iwCommit lock held
>   protected void openWriter() throws IOException {
> if (writer==null) {
>   writer = createMainIndexWriter("DirectUpdateHandler2", false);
> }
>   }
> 
> addDoc(...) {
> ...
> 
> iwAccess.lock();
> try {
> 
>   // We can't use iwCommit to protect internal data here, since it would
>   // block other addDoc calls.  Hence, we synchronize to protect
> internal
>   // state.  This is safe as all other state-changing operations are
>   // protected with iwCommit (which iwAccess excludes from this block).
>   synchronized (this) {
> // adding document -- prep writer
> closeSearcher();
> openWriter();
> tracker.addedDocument();
>   } // end synchronized block
> 
>   // this is the only unsynchronized code in the iwAccess block, which
>   // should account for most of the time
> 
> }
> 
> commit() {
> ...
> 
> iwCommit.lock();
> try {
>   log.info("start "+cmd);
> 
>   if (cmd.optimize) {
> closeSearcher();
> openWriter();
> writer.optimize(cmd.maxOptimizeSegments);
>   }
> 
>   closeSearcher();
>   closeWriter();
> 
>   callPostCommitCallbacks();
> }
> 
> Thanks
> Bharat Jain
> 

Re: how to normalize a query

[Lance Norskog]

In the general case, this would require a new method on compound queries 
to sort themselves into a "canonical" order, or refuse to. Somehow, I 
don't think this will happen. However, it could be done with boolean 
queries only, which would make it somewhat easier to combinatorically 
compose OR filter queries from a base set.


Jonathan Rochkind wrote:
Those two queries might NOT always be 'the same', depending on how you 
have your Solr request handler set up.


For instance, if you have dismax with a ps boost, then "two one" may 
end up with different relevancy scores than "one two", because the 
query as a phrase will be used for boosting, and those are two 
different phrases.  That isn't the only example.


In general, I think this isn't a Solr issue, it depends on the 
semantics of your app and setup, and you've just got to write code in 
the language of your choice to determine if two q strings are 'the 
same' for your purposes.


Jonathan

dan sutton wrote:
What I wanted was a was to determine that simply the query q="one 
two" is

equivalent to q="two one", by normalizing I might have

q="one two" for both for example, and then the q.hashCode() would be the
same

Simply using q.hashCode()  returns different values for each query 
above so

this is not suitable

Cheers
Dan

On Thu, Sep 9, 2010 at 3:36 PM, Markus Jelsma 
wrote:



LuceneQParser

http://lucene.apache.org/java/2_4_0/queryparsersyntax.html#Proximity%20Searches 



DismaxQParser
http://wiki.apache.org/solr/DisMaxQParserPlugin#qs_.28Query_Phrase_Slop.29 




On Thursday 09 September 2010 15:08:41 dan sutton wrote:

Hi,

Does anyone know how I might normalized a query so that e.g. q="one 
two"

equals q="two one"

Cheers,
Dan


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Indexing checksum of field value

[Lance Norskog]

Look at Deduplication:

http://wiki.apache.org/solr/Deduplication

It implements a unique hashcode (Lookup3Signature 
 ) as a tool that avoids 
rewriting the same document over and over. It declares this in 
solrconfig.xml instead of schema.xml.


Lance

Staffan wrote:

Hi,

I am looking for a way to store the checksum of a field's value, something like:




...


I haven't found anything like that in the docs or on google. Did I
miss something? If not, would a custom tokenizer be a good way to
implement it?

/Staffan
   

Re: Distance sorting with spatial filtering

[Lance Norskog]

I just checked out the trunk, and branch 3.x This query is accepted on 
both, but gives no responses:

http://localhost:8983/solr/select/?q=*:*&sort=dist(2,x_dt,y_dt,0,0)+asc

x_dt and y_dt are wildcard fields with the tdouble type. "tdouble" 
explicitly says it is stored and indexed. Your 'longitude' and 
'latitude' fields may not be stored?


Also, this is accepted on both branches:
http://localhost:8983/solr/select/?q=*:*&sort=sum(1)+asc

The documentation for sum() does not mention single-argument calls.

(The name Solr 1.5 is dead- the next will probably be 3.1.)

Scott K wrote:

According to the documentation, sorting by function has been a feature
since Solr 1.5. It seems like a major regression if this no longer
works.
http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function

The _val_ trick does not seem to work if used with a query term,
although I can try some more things to give 0 value to the query term.

On Wed, Sep 8, 2010 at 22:21, Lance Norskog  wrote:
   

It says that the field "sum(1)" is not indexed. You don't have a field
called 'sum(1)'. I know there has been a lot of changes in query parsing,
and sorting by functions may be on the list. But the _val_ trick is the
older one and, and you noted, still works. The _val_ trick sets the ranking
value to the output of the function, thus indirectly doing what sort= does.

Lance

Scott K wrote:
 

I get the error on all functions.
GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
Error 400 can not sort on unindexed field: sum(1)

I tried another nightly build from today, Sep 7th, with the same
results. I attached the schema.xml

Thanks for the help!
Scott

On Wed, Sep 1, 2010 at 18:43, Lance Norskogwrote:

   

Post your schema.

On Mon, Aug 30, 2010 at 2:04 PM, Scott Kwrote:

 

The new spatial filtering (SOLR-1586) works great and is much faster
than fq={!frange. However, I am having problems sorting by distance.
If I try
GET
'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
I get an error:
Error 400 can not sort on unindexed field:
dist(2,latitude,longitude,0,0)

I was able to work around this with
GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
latitude, longitude, 0,0),1,1,1)"&fl=*,score'

But why isn't sorting by functions working? I get this error with any
function I try to sort on.This is a nightly trunk build from Aug 25th.
I see SOLR-1297 was reopened, but that seems to be for edge cases.

Second question: I am using the LatLonType from the Spatial Filtering
wiki, http://wiki.apache.org/solr/SpatialSearch
Are there any distance sorting functions that use this field, or do I
need to have three indexed fields, store_lat_lon, latitude, and
longitude, if I want both filtering and sorting by distance.

Thanks, Scott


   


--
Lance Norskog
goks...@gmail.com


 
 

Re: Distance sorting with spatial filtering

[Scott K]

On Thu, Sep 9, 2010 at 21:00, Lance Norskog  wrote:
> I just checked out the trunk, and branch 3.x This query is accepted on both,
> but gives no responses:
> http://localhost:8983/solr/select/?q=*:*&sort=dist(2,x_dt,y_dt,0,0)+asc

So you are saying when you add the sort parameter you get no results
back, but do not get the error I am seeing? Should I open a Jira
ticket?

> x_dt and y_dt are wildcard fields with the tdouble type. "tdouble"
> explicitly says it is stored and indexed. Your 'longitude' and 'latitude'
> fields may not be stored?

No, they are stored.
http://localhost:8983/solr/select?q=*:*&rows=1&wt=xml&indent=true



  0
  9


  
...
47.6636
-122.3054


> Also, this is accepted on both branches:
> http://localhost:8983/solr/select/?q=*:*&sort=sum(1)+asc
>
> The documentation for sum() does not mention single-argument calls.

This also fails
http://localhost:8983/solr/select/?q=*:*&sort=sum(1,2)+asc
http://localhost:8983/solr/select/?q=*:*&sort=sum(latitude,longitude)+asc


> Scott K wrote:
>>
>> According to the documentation, sorting by function has been a feature
>> since Solr 1.5. It seems like a major regression if this no longer
>> works.
>> http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
>>
>> The _val_ trick does not seem to work if used with a query term,
>> although I can try some more things to give 0 value to the query term.
>>
>> On Wed, Sep 8, 2010 at 22:21, Lance Norskog  wrote:
>>
>>>
>>> It says that the field "sum(1)" is not indexed. You don't have a field
>>> called 'sum(1)'. I know there has been a lot of changes in query parsing,
>>> and sorting by functions may be on the list. But the _val_ trick is the
>>> older one and, and you noted, still works. The _val_ trick sets the
>>> ranking
>>> value to the output of the function, thus indirectly doing what sort=
>>> does.
>>>
>>> Lance
>>>
>>> Scott K wrote:
>>>

 I get the error on all functions.
 GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
 Error 400 can not sort on unindexed field: sum(1)

 I tried another nightly build from today, Sep 7th, with the same
 results. I attached the schema.xml

 Thanks for the help!
 Scott

 On Wed, Sep 1, 2010 at 18:43, Lance Norskog    wrote:


>
> Post your schema.
>
> On Mon, Aug 30, 2010 at 2:04 PM, Scott K    wrote:
>
>
>>
>> The new spatial filtering (SOLR-1586) works great and is much faster
>> than fq={!frange. However, I am having problems sorting by distance.
>> If I try
>> GET
>>
>> 'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
>> I get an error:
>> Error 400 can not sort on unindexed field:
>> dist(2,latitude,longitude,0,0)
>>
>> I was able to work around this with
>> GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
>> latitude, longitude, 0,0),1,1,1)"&fl=*,score'
>>
>> But why isn't sorting by functions working? I get this error with any
>> function I try to sort on.This is a nightly trunk build from Aug 25th.
>> I see SOLR-1297 was reopened, but that seems to be for edge cases.
>>
>> Second question: I am using the LatLonType from the Spatial Filtering
>> wiki, http://wiki.apache.org/solr/SpatialSearch
>> Are there any distance sorting functions that use this field, or do I
>> need to have three indexed fields, store_lat_lon, latitude, and
>> longitude, if I want both filtering and sorting by distance.
>>
>> Thanks, Scott
>>
>>
>>
>
> --
> Lance Norskog
> goks...@gmail.com
>
>
>
>>>
>>>
>

How to Update Value of One Field of a Document in Index?

[Savannah Beckett]

I use nutch to crawl and index to Solr.  My code is working.  Now, I want to 
update the value of one of the fields of a document in the solr index after the 
document was already indexed, and I have only the document id.  How do I do 
that?  

Thanks.


  

Re: How to extend IndexSchema and SchemaField

[Javier Diaz]

Hi Renaud:

Right now we faced the same issue: while developing a new RequestHandler we
needed to add some information per field (in our case to define some fields
as defaultSearchFields). We thought that this could be added externally in a
configuration file or in the schema as new attributes.

Looking at the code we found out that there's no way to extend the schema.
Finally we copied part of the code that reads the schema in our
RequestHandler. It works but I'm not sure if it's the best way to do it. Let
me know if you want our code as an example.

I do think its worth to open an issue for this requirement.

Regards,

Javier

On Thu, Sep 9, 2010 at 7:29 PM, Renaud Delbru wrote:

>  Hi,
>
> so I suppose there is no solution. Is there a chance that SchemaField
> becomes extensible in the future ? Because, at the moment, all the field
> attributes (indexed, stored, etc.) are hardcoded inside SchemaField. Do you
> think it is worth opening an issue about it ?
> --
> Renaud Delbru
>
> On 07/09/10 16:13, Renaud Delbru wrote:
>
>>  Hi,
>>
>> I would like to extend the field node in the schema.xml by adding new
>> attributes. For example, I would like to be able to write:
>> 
>> And be able to access myattribute directly from IndexSchema and
>> SchemaField objects. However, these two classes are final, and also not very
>> easy to extend ?
>> Is there any other solutions ?
>>
>> thanks,
>>
>
>

Re: How to Update Value of One Field of a Document in Index?

[Grijesh.singh]

There is no way to update any field in solr,You have to reindex that entire
document again.

you can get that doc from index create xml with existing fields with your
updated field and post that xml to solr.

-
Grijesh
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Update-Value-of-One-Field-of-a-Document-in-Index-tp1450648p1450772.html
Sent from the Solr - User mailing list archive at Nabble.com.