Re: uuid, COMB uuid, distributed farms

[Dennis Gearon]

Right, Lance, I meant in the field defintion.

I appreciate your help and direction.

 Dennis Gearon


Signature Warning

It is always a good idea to learn from your own mistakes. It is usually a 
better 
idea to learn from others’ mistakes, so you do not have to make them yourself. 
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'


EARTH has a Right To Life,
otherwise we all die.



- Original Message 
From: Lance Norskog 
To: solr-user@lucene.apache.org
Sent: Tue, January 4, 2011 7:15:07 PM
Subject: Re: uuid, COMB uuid, distributed farms

'NOT NULL' in the schema is 'required=true' in a  element.
'Search for NOT NULL' is a little odd: you search for a range and then
negate the search, meaning for documents with nothing in that field.
This standard query does it:
-field:[* TO *]

On Tue, Jan 4, 2011 at 2:49 PM, Dennis Gearon  wrote:
> Thanks Lance.
>
> I will be generating the COMB style of UUID external to Solr.
> Prevents a lot of index paging during INSERTS on DBs, maby eSolr too.
>
> So I would not use 'NEW' in the following, right?
> Just leave default out?
> Some sort of NOT NULL available in a Solr Schema?
>
> 
> PHP code to make the COMB style of UUID,
> easily adapted to other languages, some solutions already exist:
>
> 
> //requires php5_uuid module in PHP
> function make_comb_uuid(){
>  uuid_create(&$v4);
>  uuid_make($v4, UUID_MAKE_V4);
>  uuid_export($v4, UUID_FMT_STR, &$v4String);
>  $var=gettimeofday();
>  return
> substr($v4String,0,24).substr(dechex($var['sec'].$var['usec']),0,12);
>
> }
>
> 
>
> Dennis Gearon
>
>
>
>
> - Original Message 
> From: Lance Norskog 
> To: solr-user@lucene.apache.org
> Sent: Tue, January 4, 2011 2:15:32 PM
> Subject: Re: uuid, COMB uuid, distributed farms
>
> http://wiki.apache.org/solr/UniqueKey
>
> On Mon, Jan 3, 2011 at 6:55 PM, pankaj bhatt  wrote:
>> HI Dennis,
>>  I have used UUID in context of an application where an installation id
>> (UUID) is generated by the code. It caters to around 10K users.
>>  I have not used it in context of SOLR.
>>
>> / Pankaj Bhatt.
>>
>> On Mon, Jan 3, 2011 at 11:05 PM, Dennis Gearon wrote:
>>
>>> Thank you Pankaj.
>>>
>>> How large was your installation of Solr? I'm hoping to get mine to be
>>> multinational and making plans for that as I go. So having unique ids,
>>> UUIDs,
>>> that cover a huge addressable space is a requirement.
>>>
>>> If your's was comparable, how were your replication issues, merging issues,
>>> anthing else related to getting large datasets searchable and unique?
>>>
>>>  Dennis Gearon
>>>
>>>
>>> Signature Warning
>>> 
>>> It is always a good idea to learn from your own mistakes. It is usually a
>>> better
>>> idea to learn from others’ mistakes, so you do not have to make them
>>> yourself.
>>> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>>
>>>
>>> EARTH has a Right To Life,
>>> otherwise we all die.
>>>
>>>
>>>
>>> - Original Message 
>>> From: pankaj bhatt 
>>> To: solr-user@lucene.apache.org; gear...@sbcglobal.ne
>>> Sent: Mon, January 3, 2011 8:55:21 AM
>>> Subject: Re: uuid, COMB uuid, distributed farms
>>>
>>> Hi Dennis,
>>>
>>>I have used UUID's in my project to identify a basic installation of
>>> the client.
>>>Can i be of any help.
>>>
>>> / Pankaj Bhatt.
>>>
>>> On Mon, Jan 3, 2011 at 3:28 AM, Dennis Gearon 
>>> wrote:
>>>
>>> > Planning ahead here.
>>> >
>>> > Anyone have experience with UUIDs, COMB UUIDs (sequential) in large,
>>> > internatiionally distributed Solr/Database project.
>>> >
>>> >  Dennis Gearon
>>> >
>>> >
>>> > Signature Warning
>>> > 
>>> > It is always a good idea to learn from your own mistakes. It is usually a
>>> > better
>>> > idea to learn from others’ mistakes, so you do not have to make them
>>> > yourself.
>>> > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>> >
>>> >
>>> > EARTH has a Right To Life,
>>> > otherwise we all die.
>>> >
>>> >
>>>
>>>
>>
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>
>



-- 
Lance Norskog
goks...@gmail.com

Re: Error when running simple queries to solr

[Stevenson Cunanan]

Hi,

Thanks guys for the quick reply. I really appreciate it.
in reply to Lance, I'm using apache-solr-1.4.1.

My /opt/solr directory contains the following:

   - drwxr-xr-x 10 tomcat6 tomcat6 4096 2011-01-04 14:44 .
   - drwxr-xr-x  4 rootroot4096 2011-01-04 09:53 ..
   - lrwxrwxrwx  1 tomcat6 tomcat6   53 2011-01-04 14:44 apache-solr.war ->
   download/apache-solr-1.4.1/dist/apache-solr-1.4.1.war
   - -rw-r--r--  1 tomcat6 tomcat6  517 2011-01-03 11:05 build.xml
   - drwxr-xr-x  4 tomcat6 tomcat6 4096 2011-01-04 14:43 conf
   - drwxr-xr-x  5 tomcat6 tomcat6 4096 2011-01-04 09:53 data
   - drwxr-xr-x  3 tomcat6 tomcat6 4096 2011-01-04 10:33 dist
   - drwxr-xr-x  3 tomcat6 tomcat6 4096 2011-01-04 10:33 download
   - drwxr-xr-x  3 tomcat6 tomcat6 4096 2011-01-04 10:33 lib
   - drwxr-xr-x  2 tomcat6 tomcat6 4096 2011-01-04 10:15 scripts


my data/index file contains nothing so there is no index.
i understand that I need to recreate indexes using 2 curls
curl -v
http://localhost:8080/asolr/update?stream.body=%3Cdelete%3E%3Cquery%3E*:
curl -v http://localhost:8080/asolr/update?stream.body=%3Ccommit/%3E

however my curls fail with the same "no segments* file found ..." error.

I edited post.sh in the example and saw that i calls
localhost:8983/solr/update. From this i assumed that i should used similar
calls to create the indexes. Am I correct with this?

Best Regards,

Stevenson Lee

On Wed, Jan 5, 2011 at 11:19 AM, Lance Norskog  wrote:

> Which Solr release is this?
>
> There is supposed to be a core/data directory, and inside that
> core/data/index with the index files. If you have no index, just make
> core/data and Solr will make core/data/index.
>
> LanceI
>
> On Tue, Jan 4, 2011 at 6:45 PM, Stevenson Cunanan
>  wrote:
> > Hi,
> >
> > I'm very new to solr and am still reading through the wiki. I am running
> a
> > custom solr project made by colleague for our search; after placing it on
> > opt/solr and adding the tomcat tomcat context fragment to point to
> docbase.
> > I basically followed the instructions on
> > http://wiki.apache.org/solr/SolrTomcat.
> >
> > I did a test just to see if my setup worked. I did a curl
> >
> > curl -s 'http://localhost:8080/asolr/select?q=boo&fl=id&wt=json'
> >
> > but i get a tomcat error. Do any of you have an idea?
> >
> > Apache Tomcat/6.0.24 - Error
> report
> > HTTP Status 500 - Severe errors in solr configuration.
> >
> > Check your log files for more detailed information on what may be wrong.
> >
> > If you want solr to continue after configuration errors, change:
> >
> >  false
> >
> > in null
> >
> > -
> > java.lang.RuntimeException: java.io.FileNotFoundException: no segments*
> file
> > found in org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index:
> files:
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
> > at org.apache.solr.core.SolrCore.(SolrCore.java:579)
> > at
> >
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422)
> > at
> >
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115)
> > at
> >
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838)
> > at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4488)
> > at
> >
> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3228)
> > at
> >
> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:918)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:544)
> > at
> >
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:121)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > at
> >
> org.apache.catalina.core.Appli

Re: [sqljdbc4.jar] Errors

[Estrada Groups]

I downloaded that driver today and will test it tomorrow. Thanks for the tip! 
Would you mind sending an XML code snippet if it's any different to load than 
the MS driver?

Thanks,

Adam 

Sent from my iPhone

On Jan 4, 2011, at 11:34 PM, Gora Mohanty  wrote:

> On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
>  wrote:
>> Can anyone help me with the following error. I upgraded my database to SQL
>> Server 2008 SP2 and now I get the following error. It was working with SQL
>> Server 2005.
>> 
>> 
>> Caused by: java.lang.UnsupportedOperationException: Java Runtime Environment
>> (JR
>> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
>> libra
>> ry, which provides support for JDBC 4.0.
> [...]
> 
> You need to do pretty much what the error message says. The last
> time we looked at MS SQL Server, the Microsoft JDBC driver had
> both a sqljdbc.jar, and a sqljdbc4.jar. The former was for Java 1.5,
> and the latter for Java 1.6.
> 
> You might also consider moving to the open source jTDS driver
> ( http://jtds.sourceforge.net/ ). We moved to it in the advice of
> some people on this list, and it does seem to perform better, drop
> connections less often, etc.
> 
> Regards,
> Gora

Re: [sqljdbc4.jar] Errors

[Gora Mohanty]

On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
 wrote:
> Can anyone help me with the following error. I upgraded my database to SQL
> Server 2008 SP2 and now I get the following error. It was working with SQL
> Server 2005.
>
> 
> Caused by: java.lang.UnsupportedOperationException: Java Runtime Environment
> (JR
> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
> libra
> ry, which provides support for JDBC 4.0.
[...]

You need to do pretty much what the error message says. The last
time we looked at MS SQL Server, the Microsoft JDBC driver had
both a sqljdbc.jar, and a sqljdbc4.jar. The former was for Java 1.5,
and the latter for Java 1.6.

You might also consider moving to the open source jTDS driver
( http://jtds.sourceforge.net/ ). We moved to it in the advice of
some people on this list, and it does seem to perform better, drop
connections less often, etc.

Regards,
Gora

Re: How to integrate Solr with C/CPP client

[Gora Mohanty]

On Wed, Jan 5, 2011 at 9:54 AM, prasad deshpande
 wrote:
> It seems Solr/Lucene is completely developed in Java.
> I want to integrate Solr with C/CPP client. Is it possible? If yes, how?
[]

Lucene has a C port called Lucy ( http://incubator.apache.org/lucy/ )

Solr can be accessed as a web service, so it is possible to use a
C/C++ client.

Regards,
Gora

How to integrate Solr with C/CPP client

[prasad deshpande]

It seems Solr/Lucene is completely developed in Java.
I want to integrate Solr with C/CPP client. Is it possible? If yes, how?

Thanks;
Prasad

Re: [sqljdbc4.jar] Errors

[Adam Estrada]

I got the latest jar file from the MS website and then changed the
authentication to Mixed Mode on my DB. That seems to have fixed it. My 2005
Server was Windows Authentication only and that worked so there are
obviously quite a few differences between the versions of the DB. I learn
something new every day

Thanks for the feedback!
Adam

On Tue, Jan 4, 2011 at 10:20 PM, Lance Norskog  wrote:

> Do you get a new JDBC driver jar with 2008? Look around the
> distribution or the MS web site.
>
> On Tue, Jan 4, 2011 at 7:06 PM, pankaj bhatt  wrote:
> > Hi Adam,
> >   Can you try by downgrading your Java version to java 5.
> > However i am using Java 6u13 with sqljdbc4.jar , i however do not
> > get any error.
> > If possible, can you pleease also try with some other version of
> > Java 6.
> >
> > / Pankaj Bhatt.
> >
> > On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
> > wrote:
> >
> >> Can anyone help me with the following error. I upgraded my database to
> SQL
> >> Server 2008 SP2 and now I get the following error. It was working with
> SQL
> >> Server 2005.
> >>
> >> 
> >> Caused by: java.lang.UnsupportedOperationException: Java Runtime
> >> Environment
> >> (JR
> >> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar
> class
> >> libra
> >> ry, which provides support for JDBC 4.0.
> >>
> >> Any tips on this would be great!
> >>
> >> Thanks,
> >> Adam
> >>
> >
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>

Re: [sqljdbc4.jar] Errors

[Lance Norskog]

Do you get a new JDBC driver jar with 2008? Look around the
distribution or the MS web site.

On Tue, Jan 4, 2011 at 7:06 PM, pankaj bhatt  wrote:
> Hi Adam,
>           Can you try by downgrading your Java version to java 5.
>         However i am using Java 6u13 with sqljdbc4.jar , i however do not
> get any error.
>         If possible, can you pleease also try with some other version of
> Java 6.
>
> / Pankaj Bhatt.
>
> On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
> wrote:
>
>> Can anyone help me with the following error. I upgraded my database to SQL
>> Server 2008 SP2 and now I get the following error. It was working with SQL
>> Server 2005.
>>
>> 
>> Caused by: java.lang.UnsupportedOperationException: Java Runtime
>> Environment
>> (JR
>> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
>> libra
>> ry, which provides support for JDBC 4.0.
>>
>> Any tips on this would be great!
>>
>> Thanks,
>> Adam
>>
>



-- 
Lance Norskog
goks...@gmail.com

Re: Error when running simple queries to solr

[Lance Norskog]

Which Solr release is this?

There is supposed to be a core/data directory, and inside that
core/data/index with the index files. If you have no index, just make
core/data and Solr will make core/data/index.

LanceI

On Tue, Jan 4, 2011 at 6:45 PM, Stevenson Cunanan
 wrote:
> Hi,
>
> I'm very new to solr and am still reading through the wiki. I am running a
> custom solr project made by colleague for our search; after placing it on
> opt/solr and adding the tomcat tomcat context fragment to point to docbase.
> I basically followed the instructions on
> http://wiki.apache.org/solr/SolrTomcat.
>
> I did a test just to see if my setup worked. I did a curl
>
> curl -s 'http://localhost:8080/asolr/select?q=boo&fl=id&wt=json'
>
> but i get a tomcat error. Do any of you have an idea?
>
> Apache Tomcat/6.0.24 - Error report
> HTTP Status 500 - Severe errors in solr configuration.
>
> Check your log files for more detailed information on what may be wrong.
>
> If you want solr to continue after configuration errors, change:
>
>  false
>
> in null
>
> -
> java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file
> found in org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index: files:
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
> at org.apache.solr.core.SolrCore.(SolrCore.java:579)
> at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83)
> at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295)
> at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422)
> at
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115)
> at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4488)
> at
> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3228)
> at
> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:918)
> at
> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:544)
> at
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:121)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:558)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: no segments* file found in
> org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index: files:
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:634)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:403)
> at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057)
> ... 27 more
> type Status
> reportmessage Severe errors in solr configurati

Re: abort data import on errors

[Lance Norskog]

The DataImportHandler does this already. In fact you can set what
happens on an indexing error.

On Tue, Jan 4, 2011 at 4:59 PM, Tri Nguyen  wrote:
> I didn't want to issue the rollback command but have solr automatically 
> detect exceptions and rollback should there be exceptions.
>
> Probably there's an attribute I can configure to specify this for solr to 
> understand.
>
> Tri
>
> --- On Tue, 1/4/11, Markus Jelsma  wrote:
>
>
> From: Markus Jelsma 
> Subject: Re: abort data import on errors
> To: solr-user@lucene.apache.org
> Date: Tuesday, January 4, 2011, 4:57 PM
>
>
> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>
>> Hi,
>>
>> Is there a way to specify to abort (rollback) the data import should there
>> be an error/exception?
>> If everything runs smoothly, commit the data import.
>>
>> Thanks,
>>
>> Tri
>



-- 
Lance Norskog
goks...@gmail.com

Re: uuid, COMB uuid, distributed farms

[Lance Norskog]

'NOT NULL' in the schema is 'required=true' in a  element.
'Search for NOT NULL' is a little odd: you search for a range and then
negate the search, meaning for documents with nothing in that field.
This standard query does it:
-field:[* TO *]

On Tue, Jan 4, 2011 at 2:49 PM, Dennis Gearon  wrote:
> Thanks Lance.
>
> I will be generating the COMB style of UUID external to Solr.
> Prevents a lot of index paging during INSERTS on DBs, maby eSolr too.
>
> So I would not use 'NEW' in the following, right?
> Just leave default out?
> Some sort of NOT NULL available in a Solr Schema?
>
> 
> PHP code to make the COMB style of UUID,
> easily adapted to other languages, some solutions already exist:
>
> 
> //requires php5_uuid module in PHP
> function make_comb_uuid(){
>      uuid_create(&$v4);
>      uuid_make($v4, UUID_MAKE_V4);
>      uuid_export($v4, UUID_FMT_STR, &$v4String);
>      $var=gettimeofday();
>      return
> substr($v4String,0,24).substr(dechex($var['sec'].$var['usec']),0,12);
>
> }
>
> 
>
> Dennis Gearon
>
>
>
>
> - Original Message 
> From: Lance Norskog 
> To: solr-user@lucene.apache.org
> Sent: Tue, January 4, 2011 2:15:32 PM
> Subject: Re: uuid, COMB uuid, distributed farms
>
> http://wiki.apache.org/solr/UniqueKey
>
> On Mon, Jan 3, 2011 at 6:55 PM, pankaj bhatt  wrote:
>> HI Dennis,
>>      I have used UUID in context of an application where an installation id
>> (UUID) is generated by the code. It caters to around 10K users.
>>      I have not used it in context of SOLR.
>>
>> / Pankaj Bhatt.
>>
>> On Mon, Jan 3, 2011 at 11:05 PM, Dennis Gearon wrote:
>>
>>> Thank you Pankaj.
>>>
>>> How large was your installation of Solr? I'm hoping to get mine to be
>>> multinational and making plans for that as I go. So having unique ids,
>>> UUIDs,
>>> that cover a huge addressable space is a requirement.
>>>
>>> If your's was comparable, how were your replication issues, merging issues,
>>> anthing else related to getting large datasets searchable and unique?
>>>
>>>  Dennis Gearon
>>>
>>>
>>> Signature Warning
>>> 
>>> It is always a good idea to learn from your own mistakes. It is usually a
>>> better
>>> idea to learn from others’ mistakes, so you do not have to make them
>>> yourself.
>>> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>>
>>>
>>> EARTH has a Right To Life,
>>> otherwise we all die.
>>>
>>>
>>>
>>> - Original Message 
>>> From: pankaj bhatt 
>>> To: solr-user@lucene.apache.org; gear...@sbcglobal.ne
>>> Sent: Mon, January 3, 2011 8:55:21 AM
>>> Subject: Re: uuid, COMB uuid, distributed farms
>>>
>>> Hi Dennis,
>>>
>>>        I have used UUID's in my project to identify a basic installation of
>>> the client.
>>>        Can i be of any help.
>>>
>>> / Pankaj Bhatt.
>>>
>>> On Mon, Jan 3, 2011 at 3:28 AM, Dennis Gearon 
>>> wrote:
>>>
>>> > Planning ahead here.
>>> >
>>> > Anyone have experience with UUIDs, COMB UUIDs (sequential) in large,
>>> > internatiionally distributed Solr/Database project.
>>> >
>>> >  Dennis Gearon
>>> >
>>> >
>>> > Signature Warning
>>> > 
>>> > It is always a good idea to learn from your own mistakes. It is usually a
>>> > better
>>> > idea to learn from others’ mistakes, so you do not have to make them
>>> > yourself.
>>> > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>> >
>>> >
>>> > EARTH has a Right To Life,
>>> > otherwise we all die.
>>> >
>>> >
>>>
>>>
>>
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>
>



-- 
Lance Norskog
goks...@gmail.com

Re: [sqljdbc4.jar] Errors

[pankaj bhatt]

Hi Adam,
   Can you try by downgrading your Java version to java 5.
 However i am using Java 6u13 with sqljdbc4.jar , i however do not
get any error.
 If possible, can you pleease also try with some other version of
Java 6.

/ Pankaj Bhatt.

On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
wrote:

> Can anyone help me with the following error. I upgraded my database to SQL
> Server 2008 SP2 and now I get the following error. It was working with SQL
> Server 2005.
>
> 
> Caused by: java.lang.UnsupportedOperationException: Java Runtime
> Environment
> (JR
> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
> libra
> ry, which provides support for JDBC 4.0.
>
> Any tips on this would be great!
>
> Thanks,
> Adam
>

Error when running simple queries to solr

[Stevenson Cunanan]

Hi,

I'm very new to solr and am still reading through the wiki. I am running a
custom solr project made by colleague for our search; after placing it on
opt/solr and adding the tomcat tomcat context fragment to point to docbase.
I basically followed the instructions on
http://wiki.apache.org/solr/SolrTomcat.

I did a test just to see if my setup worked. I did a curl

curl -s 'http://localhost:8080/asolr/select?q=boo&fl=id&wt=json'

but i get a tomcat error. Do any of you have an idea?

Apache Tomcat/6.0.24 - Error report
HTTP Status 500 - Severe errors in solr configuration.

Check your log files for more detailed information on what may be wrong.

If you want solr to continue after configuration errors, change:

 false

in null

-
java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file
found in org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index: files:
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
at org.apache.solr.core.SolrCore.(SolrCore.java:579)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422)
at
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4488)
at
org.apache.catalina.core.StandardContext.reload(StandardContext.java:3228)
at
org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:918)
at
org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:544)
at
org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:558)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: no segments* file found in
org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index: files:
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:634)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:403)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057)
... 27 more
type Status
reportmessage Severe errors in solr configuration.

Check your log files for more detailed information on what may be wrong.

If you want solr to continue after configuration errors, change:

 false

in null

-
java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file
found in org.apache.lucene.store.NIOFSDirectory@/opt/solr/data/index: files:
at org.apache.solr.core.SolrCore.g

Re: DataImportHanlder - Multiple entities will step into each other

[yu shen]

Very detailed explanation, will look into it. Thanks.

2011/1/5 Erick Erickson 

> No, this is all #explicitly# defined in schema.xml. Solr has no required
> fields.
> If you're using the default schema.xml, the problem is that that file
> defines
> the id field with 'required="true"' option set, so any doc that does not
> have an
> id field is rejected.
>
> Id is used in the default schema in conjunction with
> id to enforce document uniqueness.
> Solr only uses  to identify documents for replacement. That
> is, if you add a document with a  that is already in the index,
> the old version of the document is removed and the new one added.
>
> But  is not required at all and no field in solr is required
> unless
> required="true" is specified.
>
> Best
> Erick
>
> On Tue, Jan 4, 2011 at 7:19 PM, yu shen  wrote:
>
> > I tried the sql way, and not work as expected.
> > According to my experiments, id is an implicit required field of solr. If
> I
> > change id to table_id, and add field definition in schema.xml, while data
> > importing, there will be an error reported.
> >
> > Please correct me if I am wrong.
> >
> > 2011/1/5 Lance Norskog 
> >
> > > This SQL syntax should do it: "select id, field as table_id, field".
> > >
> > > On Tue, Jan 4, 2011 at 5:59 AM, yu shen  wrote:
> > > > Thanks for the prompt reply. Let me try. Delete is not a big deal for
> > the
> > > > moment.
> > > >
> > > > 2011/1/4 Matti Oinas 
> > > >
> > > >> I managed to do that by using TemplateTransformer
> > > >>
> > > >> 
> > > >>  
> > > >>  />
> > > >> ...
> > > >>  
> > > >> 
> > > >> ...
> > > >> 
> > > >>
> > > >> Only problem is that delta import fails to perform delete to the
> > > >> index. It seems that TemplateTransformer is not used when performing
> > > >> delete so delete by id doesn't work.
> > > >>
> > > >>
> > > >>
> > > >> 2011/1/4 yu shen :
> > > >> > Hi All,
> > > >> >
> > > >> > I have a dataimporthandler config file as below. It contains
> > multiple
> > > >> > entities:
> > > >> > 
> > > >> > > > >> >
> > > >> >
> > > >>
> > >
> >
> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
> > > >> > />
> > > >> >
> > > >> > > > query="...">
> > > >> > > > query="">
> > > >> >
> > > >> >
> > > >> > 
> > > >> >
> > > >> > All data are from a database. Problem is item/company and other
> > entity
> > > >> all
> > > >> > have the field 'id', with value start from 1 to n. In this case,
> > > >> > item/company etc. will step into each other.
> > > >> > Is there a way to prevent is from happening. Such as designate
> > > different
> > > >> > entity to different partition.
> > > >> >
> > > >> > One way I can think of is to seperate different entity to
> different
> > > >> > instance, which is not ideal solution IMO.
> > > >> >
> > > >> > Would some one point me to a reference? And also give some
> > > instructions?
> > > >> >
> > > >>
> > > >
> > >
> > >
> > >
> > > --
> > > Lance Norskog
> > > goks...@gmail.com
> > >
> >
>

Re: DataImportHanlder - Multiple entities will step into each other

[Erick Erickson]

No, this is all #explicitly# defined in schema.xml. Solr has no required
fields.
If you're using the default schema.xml, the problem is that that file
defines
the id field with 'required="true"' option set, so any doc that does not
have an
id field is rejected.

Id is used in the default schema in conjunction with
id to enforce document uniqueness.
Solr only uses  to identify documents for replacement. That
is, if you add a document with a  that is already in the index,
the old version of the document is removed and the new one added.

But  is not required at all and no field in solr is required
unless
required="true" is specified.

Best
Erick

On Tue, Jan 4, 2011 at 7:19 PM, yu shen  wrote:

> I tried the sql way, and not work as expected.
> According to my experiments, id is an implicit required field of solr. If I
> change id to table_id, and add field definition in schema.xml, while data
> importing, there will be an error reported.
>
> Please correct me if I am wrong.
>
> 2011/1/5 Lance Norskog 
>
> > This SQL syntax should do it: "select id, field as table_id, field".
> >
> > On Tue, Jan 4, 2011 at 5:59 AM, yu shen  wrote:
> > > Thanks for the prompt reply. Let me try. Delete is not a big deal for
> the
> > > moment.
> > >
> > > 2011/1/4 Matti Oinas 
> > >
> > >> I managed to do that by using TemplateTransformer
> > >>
> > >> 
> > >>  
> > >> 
> > >> ...
> > >>  
> > >> 
> > >> ...
> > >> 
> > >>
> > >> Only problem is that delta import fails to perform delete to the
> > >> index. It seems that TemplateTransformer is not used when performing
> > >> delete so delete by id doesn't work.
> > >>
> > >>
> > >>
> > >> 2011/1/4 yu shen :
> > >> > Hi All,
> > >> >
> > >> > I have a dataimporthandler config file as below. It contains
> multiple
> > >> > entities:
> > >> > 
> > >> > > >> >
> > >> >
> > >>
> >
> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
> > >> > />
> > >> >
> > >> > > query="...">
> > >> > > query="">
> > >> >
> > >> >
> > >> > 
> > >> >
> > >> > All data are from a database. Problem is item/company and other
> entity
> > >> all
> > >> > have the field 'id', with value start from 1 to n. In this case,
> > >> > item/company etc. will step into each other.
> > >> > Is there a way to prevent is from happening. Such as designate
> > different
> > >> > entity to different partition.
> > >> >
> > >> > One way I can think of is to seperate different entity to different
> > >> > instance, which is not ideal solution IMO.
> > >> >
> > >> > Would some one point me to a reference? And also give some
> > instructions?
> > >> >
> > >>
> > >
> >
> >
> >
> > --
> > Lance Norskog
> > goks...@gmail.com
> >
>

RE: Custom match scoring

[Chris Hostetter]

: Yes, I already looked dismax which I'm  using for other purposes, the big
: deal for this problem is having summed only the best match of each field. In
: dismax it sum all matches on each field.

can you describe what you want in pusedo code?

what you are describing sounds exactly like using the dismax parser with 
tie=0.  that way each "clause" of the input only gets the max score from 
each of hte fields in the qf param. 

dismax doesn't sum all matches on each field, it sums the *max* match on 
each field, plus a tie breaker multiplier times the sum of all other 
matches on each field -- if tie=0 it's a true disjunction max query, if 
tie=1 it's a true disjunction sum query.

-Hoss

Re: abort data import on errors

[Tri Nguyen]

I didn't want to issue the rollback command but have solr automatically detect 
exceptions and rollback should there be exceptions.
 
Probably there's an attribute I can configure to specify this for solr to 
understand.
 
Tri

--- On Tue, 1/4/11, Markus Jelsma  wrote:


From: Markus Jelsma 
Subject: Re: abort data import on errors
To: solr-user@lucene.apache.org
Date: Tuesday, January 4, 2011, 4:57 PM


http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22

> Hi,
>  
> Is there a way to specify to abort (rollback) the data import should there
> be an error/exception? 
> If everything runs smoothly, commit the data import.
>  
> Thanks,
>  
> Tri

Re: abort data import on errors

[Markus Jelsma]

http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22

> Hi,
>  
> Is there a way to specify to abort (rollback) the data import should there
> be an error/exception? 
> If everything runs smoothly, commit the data import.
>  
> Thanks,
>  
> Tri

abort data import on errors

[Tri Nguyen]

Hi,
 
Is there a way to specify to abort (rollback) the data import should there be 
an error/exception?
 
If everything runs smoothly, commit the data import.
 
Thanks,
 
Tri

Re: DataImportHanlder - Multiple entities will step into each other

[yu shen]

I tried the sql way, and not work as expected.
According to my experiments, id is an implicit required field of solr. If I
change id to table_id, and add field definition in schema.xml, while data
importing, there will be an error reported.

Please correct me if I am wrong.

2011/1/5 Lance Norskog 

> This SQL syntax should do it: "select id, field as table_id, field".
>
> On Tue, Jan 4, 2011 at 5:59 AM, yu shen  wrote:
> > Thanks for the prompt reply. Let me try. Delete is not a big deal for the
> > moment.
> >
> > 2011/1/4 Matti Oinas 
> >
> >> I managed to do that by using TemplateTransformer
> >>
> >> 
> >>  
> >> 
> >> ...
> >>  
> >> 
> >> ...
> >> 
> >>
> >> Only problem is that delta import fails to perform delete to the
> >> index. It seems that TemplateTransformer is not used when performing
> >> delete so delete by id doesn't work.
> >>
> >>
> >>
> >> 2011/1/4 yu shen :
> >> > Hi All,
> >> >
> >> > I have a dataimporthandler config file as below. It contains multiple
> >> > entities:
> >> > 
> >> > >> >
> >> >
> >>
> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
> >> > />
> >> >
> >> > query="...">
> >> > query="">
> >> >
> >> >
> >> > 
> >> >
> >> > All data are from a database. Problem is item/company and other entity
> >> all
> >> > have the field 'id', with value start from 1 to n. In this case,
> >> > item/company etc. will step into each other.
> >> > Is there a way to prevent is from happening. Such as designate
> different
> >> > entity to different partition.
> >> >
> >> > One way I can think of is to seperate different entity to different
> >> > instance, which is not ideal solution IMO.
> >> >
> >> > Would some one point me to a reference? And also give some
> instructions?
> >> >
> >>
> >
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>

[sqljdbc4.jar] Errors

[Adam Estrada]

Can anyone help me with the following error. I upgraded my database to SQL
Server 2008 SP2 and now I get the following error. It was working with SQL
Server 2005.


Caused by: java.lang.UnsupportedOperationException: Java Runtime Environment
(JR
E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
libra
ry, which provides support for JDBC 4.0.

Any tips on this would be great!

Thanks,
Adam

Re: uuid, COMB uuid, distributed farms

[Dennis Gearon]

Thanks Lance.

I will be generating the COMB style of UUID external to Solr.
Prevents a lot of index paging during INSERTS on DBs, maby eSolr too.

So I would not use 'NEW' in the following, right? 
Just leave default out? 
Some sort of NOT NULL available in a Solr Schema?


PHP code to make the COMB style of UUID,
easily adapted to other languages, some solutions already exist:


//requires php5_uuid module in PHP
function make_comb_uuid(){
  uuid_create(&$v4);
  uuid_make($v4, UUID_MAKE_V4);
  uuid_export($v4, UUID_FMT_STR, &$v4String);
  $var=gettimeofday();
  return 
substr($v4String,0,24).substr(dechex($var['sec'].$var['usec']),0,12); 

}

 

Dennis Gearon




- Original Message 
From: Lance Norskog 
To: solr-user@lucene.apache.org
Sent: Tue, January 4, 2011 2:15:32 PM
Subject: Re: uuid, COMB uuid, distributed farms

http://wiki.apache.org/solr/UniqueKey

On Mon, Jan 3, 2011 at 6:55 PM, pankaj bhatt  wrote:
> HI Dennis,
>  I have used UUID in context of an application where an installation id
> (UUID) is generated by the code. It caters to around 10K users.
>  I have not used it in context of SOLR.
>
> / Pankaj Bhatt.
>
> On Mon, Jan 3, 2011 at 11:05 PM, Dennis Gearon wrote:
>
>> Thank you Pankaj.
>>
>> How large was your installation of Solr? I'm hoping to get mine to be
>> multinational and making plans for that as I go. So having unique ids,
>> UUIDs,
>> that cover a huge addressable space is a requirement.
>>
>> If your's was comparable, how were your replication issues, merging issues,
>> anthing else related to getting large datasets searchable and unique?
>>
>>  Dennis Gearon
>>
>>
>> Signature Warning
>> 
>> It is always a good idea to learn from your own mistakes. It is usually a
>> better
>> idea to learn from others’ mistakes, so you do not have to make them
>> yourself.
>> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>
>>
>> EARTH has a Right To Life,
>> otherwise we all die.
>>
>>
>>
>> - Original Message 
>> From: pankaj bhatt 
>> To: solr-user@lucene.apache.org; gear...@sbcglobal.ne
>> Sent: Mon, January 3, 2011 8:55:21 AM
>> Subject: Re: uuid, COMB uuid, distributed farms
>>
>> Hi Dennis,
>>
>>I have used UUID's in my project to identify a basic installation of
>> the client.
>>Can i be of any help.
>>
>> / Pankaj Bhatt.
>>
>> On Mon, Jan 3, 2011 at 3:28 AM, Dennis Gearon 
>> wrote:
>>
>> > Planning ahead here.
>> >
>> > Anyone have experience with UUIDs, COMB UUIDs (sequential) in large,
>> > internatiionally distributed Solr/Database project.
>> >
>> >  Dennis Gearon
>> >
>> >
>> > Signature Warning
>> > 
>> > It is always a good idea to learn from your own mistakes. It is usually a
>> > better
>> > idea to learn from others’ mistakes, so you do not have to make them
>> > yourself.
>> > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>> >
>> >
>> > EARTH has a Right To Life,
>> > otherwise we all die.
>> >
>> >
>>
>>
>



-- 
Lance Norskog
goks...@gmail.com

Re: Replication: the web application [/solr] .. likely to create a memory leak

[Markus Jelsma]

I don't have Windows :)

> Is this on Windows or Unix? Windows will not delete a file that is still
> open.
> 
> On Tue, Jan 4, 2011 at 10:07 AM, Markus Jelsma
> 
>  wrote:
> > Is it possible this problem has something to do with my old index files
> > not being removed? This problem only surfaces in my setup when i restart
> > with replication on the slave. I can confirm that for some reason my
> > replicated indexes get messed up only when i start restarting Tomcat
> > several times.
> > 
> > On Tuesday 04 January 2011 15:48:31 Yonik Seeley wrote:
> >> On Tue, Jan 4, 2011 at 9:34 AM, Robert Muir  wrote:
> >> >[junit] WARNING: test class left thread running:
> >> > Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
> >> 
> >> I suppose we should move MultiThreadedHttpConnectionManager to
> >> CoreContainer.
> >> 
> >> -Yonik
> >> http://www.lucidimagination.com
> > 
> > --
> > Markus Jelsma - CTO - Openindex
> > http://www.linkedin.com/in/markus17
> > 050-8536620 / 06-50258350

Re: problems with replication

[Lance Norskog]

Are the index files corrupted? Lucene includes a program CheckIndex
which will verify that the index data structures are not corrupted.

Lance Norskog

On Tue, Jan 4, 2011 at 7:50 AM, Roxana Angheluta  wrote:
> Thanks for replying.
>
> I am not able to reproduce it, it does not happen at every replication.
> I just see the errors in the logs.
>
> At this point I have multiple errors in the logs and I am trying to analyze 
> them systematically. See other examples below, this time from the slave. I 
> suspect they are related and have to do with the replication. I used to have 
> the server set-up as a repeater, now it is a simple slave.
>
> I don't really know where to start debugging, any advice is appreciated.
>
> roxana
>
>
> Jan 4, 2011 5:06:22 AM sun.reflect.GeneratedMethodAccessor6 invoke
> WARNING: Committed before 500 null
>
> org.mortbay.jetty.EofException
>        at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:767)
>        at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:562)
>        at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:898)
>        at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:528)
>        at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:619)
>        at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:573)
>        at 
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:183)
>        at 
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89)
>        at 
> org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:45)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:322)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
>        at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>        at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>        at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>        at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>        at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>        at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>        at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>
> -
>
> java.lang.IllegalStateException: Committed
>        at org.mortbay.jetty.Response.resetBuffer(Response.java:972)
>        at org.mortbay.jetty.Response.sendError(Response.java:238)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
>        at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>        at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>        at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>        at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>        at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>        at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>        at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>        at 
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>        at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>        at org.mortbay.jetty.Server.handle(Server.java:285)
>        at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>        at 
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>        at 
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>        at 
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>
> --- On Mon, 1/3/11, Grant Ingersoll  wrote:
>
>> From: Grant Ingersoll 
>> Subject: Re: problems with replication
>> To: solr-user@lucene.apache.org
>> Date: Monday, January 3, 2011, 4:20 PM
>> Are you able to reproduce this?
>> Can you give more details on what is happening?  Does
>> this happen every time you replicate?
>>
>> On Dec 29, 2010, at 3:11 PM, Roxana Angheluta wrote:
>>
>> > Hi,
>> >
>> > What can cause the following error on the master -
>> stack trace below.
>> > The slave is stuck in the middle of replication until
>> m

Re: Replication: the web application [/solr] .. likely to create a memory leak

[Lance Norskog]

Is this on Windows or Unix? Windows will not delete a file that is still open.

On Tue, Jan 4, 2011 at 10:07 AM, Markus Jelsma
 wrote:
> Is it possible this problem has something to do with my old index files not
> being removed? This problem only surfaces in my setup when i restart with
> replication on the slave. I can confirm that for some reason my replicated
> indexes get messed up only when i start restarting Tomcat several times.
>
> On Tuesday 04 January 2011 15:48:31 Yonik Seeley wrote:
>> On Tue, Jan 4, 2011 at 9:34 AM, Robert Muir  wrote:
>> >    [junit] WARNING: test class left thread running:
>> > Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
>>
>> I suppose we should move MultiThreadedHttpConnectionManager to
>> CoreContainer.
>>
>> -Yonik
>> http://www.lucidimagination.com
>
> --
> Markus Jelsma - CTO - Openindex
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
>



-- 
Lance Norskog
goks...@gmail.com

Re: DataImportHanlder - Multiple entities will step into each other

[Lance Norskog]

This SQL syntax should do it: "select id, field as table_id, field".

On Tue, Jan 4, 2011 at 5:59 AM, yu shen  wrote:
> Thanks for the prompt reply. Let me try. Delete is not a big deal for the
> moment.
>
> 2011/1/4 Matti Oinas 
>
>> I managed to do that by using TemplateTransformer
>>
>> 
>>  
>>     
>> ...
>>  
>>     
>> ...
>> 
>>
>> Only problem is that delta import fails to perform delete to the
>> index. It seems that TemplateTransformer is not used when performing
>> delete so delete by id doesn't work.
>>
>>
>>
>> 2011/1/4 yu shen :
>> > Hi All,
>> >
>> > I have a dataimporthandler config file as below. It contains multiple
>> > entities:
>> > 
>> >        > >
>> >
>> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
>> > />
>> >        
>> >                
>> >                
>> >                
>> >        
>> > 
>> >
>> > All data are from a database. Problem is item/company and other entity
>> all
>> > have the field 'id', with value start from 1 to n. In this case,
>> > item/company etc. will step into each other.
>> > Is there a way to prevent is from happening. Such as designate different
>> > entity to different partition.
>> >
>> > One way I can think of is to seperate different entity to different
>> > instance, which is not ideal solution IMO.
>> >
>> > Would some one point me to a reference? And also give some instructions?
>> >
>>
>



-- 
Lance Norskog
goks...@gmail.com

Re: uuid, COMB uuid, distributed farms

[Lance Norskog]

http://wiki.apache.org/solr/UniqueKey

On Mon, Jan 3, 2011 at 6:55 PM, pankaj bhatt  wrote:
> HI Dennis,
>      I have used UUID in context of an application where an installation id
> (UUID) is generated by the code. It caters to around 10K users.
>      I have not used it in context of SOLR.
>
> / Pankaj Bhatt.
>
> On Mon, Jan 3, 2011 at 11:05 PM, Dennis Gearon wrote:
>
>> Thank you Pankaj.
>>
>> How large was your installation of Solr? I'm hoping to get mine to be
>> multinational and making plans for that as I go. So having unique ids,
>> UUIDs,
>> that cover a huge addressable space is a requirement.
>>
>> If your's was comparable, how were your replication issues, merging issues,
>> anthing else related to getting large datasets searchable and unique?
>>
>>  Dennis Gearon
>>
>>
>> Signature Warning
>> 
>> It is always a good idea to learn from your own mistakes. It is usually a
>> better
>> idea to learn from others’ mistakes, so you do not have to make them
>> yourself.
>> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>>
>>
>> EARTH has a Right To Life,
>> otherwise we all die.
>>
>>
>>
>> - Original Message 
>> From: pankaj bhatt 
>> To: solr-user@lucene.apache.org; gear...@sbcglobal.ne
>> Sent: Mon, January 3, 2011 8:55:21 AM
>> Subject: Re: uuid, COMB uuid, distributed farms
>>
>> Hi Dennis,
>>
>>        I have used UUID's in my project to identify a basic installation of
>> the client.
>>        Can i be of any help.
>>
>> / Pankaj Bhatt.
>>
>> On Mon, Jan 3, 2011 at 3:28 AM, Dennis Gearon 
>> wrote:
>>
>> > Planning ahead here.
>> >
>> > Anyone have experience with UUIDs, COMB UUIDs (sequential) in large,
>> > internatiionally distributed Solr/Database project.
>> >
>> >  Dennis Gearon
>> >
>> >
>> > Signature Warning
>> > 
>> > It is always a good idea to learn from your own mistakes. It is usually a
>> > better
>> > idea to learn from others’ mistakes, so you do not have to make them
>> > yourself.
>> > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>> >
>> >
>> > EARTH has a Right To Life,
>> > otherwise we all die.
>> >
>> >
>>
>>
>



-- 
Lance Norskog
goks...@gmail.com

Re: Advice on Exact Matching?

[Jonathan Rochkind]

There is a hacky kind of thing that Bill Dueber figured out for using 
multiple fields and dismax to BOOST "exact" matches, but include all 
matches in the result set.


You have to duplicate your data in a second non-tokenized field. Then 
you use dismax pf to super boost matches on the non-tokenized field. 
Because 'pf' is a phrase search, you don't run into trouble with dismax 
"pre-tokenization" on white space, even though it's a field that might 
have internal-token whitespace. (Using a non-tokenized field with dismax 
qf will basically never match a result with whitespace, unless it's 
phrase-quoted in query. But pf works.).


Because it was a non-tokenized field, it only matches (and triggers the 
dismax ps super boost) if it's an exact match. And it works. You CAN 
normalize your 'exact match' field in field analysis, removing 
punctuation or normalizing whitespace or whatever, and that works too, 
doing it both at index and query time analysis.




On 1/4/2011 4:28 PM, Chris Hostetter wrote:

: I am trying to make sure that when I search for text—regardless of
: what that text is—that I get an exact match.  I'm *still* getting some
: issues, and this last mile is becoming very painful.  The solr field,
: for which I'm setting this up on, is pasted below my explanation.  I
: appreciate any help.

if you are using a TextField with some analysis components, it's
virtually impossible to get "exact" matches -- where my definition of
exact is that the query text is character for character identical to the
entire field value indexed.

is your definition of exact match different?  i assme it must be since you
are using TextField and talk about wanting to deal with whitespace between
words.  so i think you need to explain a little bit better what your
indexed data looks like, and what sample queries you expect to match that
data (and equally important: what queries should *not* match thta data,
and what data should *not* match those queries)

: If I want to find *all* Solr documents that match
: "[id]somejunk\hi[/id]" then life is instantly hell.

90% of the time when people have problems with "exact" matches it's
because of QueryParser meta characters -- characters like ":", "[" and
whitespace that the QUeryParser uses as instructions.  you can use the
"raw" QParser to have every character treated as a literal

defType=raw
q=[id]somejunk\hi[/id]

-Hoss

Re: Luke for inspecting indexes on remote solr servers?

[Stephen Boesch]

i am interested in the LukeRequestHandler   in fact having been pointed to
it will try to find a comprehensive list of solr request handlers for future
reference.

as regards the -X our linux box does not have xwindows installed for
security reasons, which is why did not try that approach.


2011/1/4 Peter Karich 

>  Am 04.01.2011 21:43, schrieb Ahmet Arslan:
> >> Is that supported?  Pointer(s)
> >> to how to do it?
> > perhaps http://wiki.apache.org/solr/LukeRequestHandler ?
>
> or via
> ssh u...@host -X
> ;-)
>

Re: Advice on Exact Matching?

[Chris Hostetter]

: I am trying to make sure that when I search for text—regardless of
: what that text is—that I get an exact match.  I'm *still* getting some
: issues, and this last mile is becoming very painful.  The solr field,
: for which I'm setting this up on, is pasted below my explanation.  I
: appreciate any help.

if you are using a TextField with some analysis components, it's 
virtually impossible to get "exact" matches -- where my definition of 
exact is that the query text is character for character identical to the 
entire field value indexed.

is your definition of exact match different?  i assme it must be since you 
are using TextField and talk about wanting to deal with whitespace between 
words.  so i think you need to explain a little bit better what your 
indexed data looks like, and what sample queries you expect to match that 
data (and equally important: what queries should *not* match thta data, 
and what data should *not* match those queries)

: If I want to find *all* Solr documents that match
: "[id]somejunk\hi[/id]" then life is instantly hell.

90% of the time when people have problems with "exact" matches it's 
because of QueryParser meta characters -- characters like ":", "[" and 
whitespace that the QUeryParser uses as instructions.  you can use the 
"raw" QParser to have every character treated as a literal

defType=raw
q=[id]somejunk\hi[/id]

-Hoss

Re: Luke for inspecting indexes on remote solr servers?

[Peter Karich]

 Am 04.01.2011 21:43, schrieb Ahmet Arslan:
>> Is that supported?  Pointer(s)
>> to how to do it?
> perhaps http://wiki.apache.org/solr/LukeRequestHandler ?

or via
ssh u...@host -X
;-)

Re: Luke for inspecting indexes on remote solr servers?

[Ahmet Arslan]

> Is that supported?  Pointer(s)
> to how to do it?

perhaps http://wiki.apache.org/solr/LukeRequestHandler ?

Re: Sub query using SOLR?

[Jonathan Rochkind]

Yeah, I don't believe there's any good way to do it in Solr 1.4.

You can make two queries, first make your 'sub' query, get back the list 
of values, then construct the second query where you do "{!field 
v=field_name} val1 OR val2 OR val3   OR valN"


Kind of a pain, and there is a maximum number of conditions you can have 
in there (1024 maybe?).


It is OFT requested behavior, and the feature on SOLR-2272 is very 
exciting to me and I think would meet a lot of needs, but I haven't 
tried it yet myself.


Jonathan

On 1/4/2011 2:03 PM, Steven A Rowe wrote:

Hi Barani,

I haven't tried it myself, but the limited JOIN functionality provided by 
SOLR-2272 sounds very similar to what you want to do:

https://issues.apache.org/jira/browse/SOLR-2272

Steve


-Original Message-
From: bbarani [mailto:bbar...@gmail.com]
Sent: Tuesday, January 04, 2011 1:27 PM
To: solr-user@lucene.apache.org
Subject: Sub query using SOLR?


Hi,

I am trying to use subquery in SOLR, is there a way to implement this
using
SOLR query syntax?

Something like

Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)

The thing I really want is to use output of one query to be the input of
another query.

Not sure if it is possible to use the query() function (function query)
for
my case..

Just want to know if ther is a better approach...

Thanks,
Barani
--
View this message in context: http://lucene.472066.n3.nabble.com/Sub-
query-using-SOLR-tp2193251p2193251.html
Sent from the Solr - User mailing list archive at Nabble.com.

Replication: abort-fetch and restarting

[Markus Jelsma]

Hi,

It seems abort-fetch nicely removes the index directory which i'm replicating 
to which is fine. Restarting, however, does not trigger the the same feature as 
the abort-fetch command does. At least, that's what my tests seems to tell me.

Shouldn't a restart of Solr nicely clean up the mess before exiting? And, 
shouldn't starting Solr also look for mess left behind by a possible sudden 
shutdown of the server at which the mess obviously cannot get cleaned?

If i now stop, clean and start my slave it will attempt to download an 
existing index. If i abort-fetch it will clean up the mess and (due to low 
interval polling) make another attempt. If i, however, restart (instead of 
abort-fetch) the old temporary directory will stay and needs to be deleted 
manually.

Cheers,
-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

RE: Sub query using SOLR?

[Steven A Rowe]

Hi Barani,

I haven't tried it myself, but the limited JOIN functionality provided by 
SOLR-2272 sounds very similar to what you want to do:

   https://issues.apache.org/jira/browse/SOLR-2272

Steve

> -Original Message-
> From: bbarani [mailto:bbar...@gmail.com]
> Sent: Tuesday, January 04, 2011 1:27 PM
> To: solr-user@lucene.apache.org
> Subject: Sub query using SOLR?
> 
> 
> Hi,
> 
> I am trying to use subquery in SOLR, is there a way to implement this
> using
> SOLR query syntax?
> 
> Something like
> 
> Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)
> 
> The thing I really want is to use output of one query to be the input of
> another query.
> 
> Not sure if it is possible to use the query() function (function query)
> for
> my case..
> 
> Just want to know if ther is a better approach...
> 
> Thanks,
> Barani
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Sub-
> query-using-SOLR-tp2193251p2193251.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Sub query using SOLR?

[Dennis Gearon]

Essentially, a subuery is an AND expression where you ask the database to find 
the identifier or set of identifiers to then use in the query outside the 
subquery.

The data that you put into a Solr index is flattened, denormalized.

So take the suquery field values and put them in an AND part of the query to 
Solr.

 Dennis Gearon


Signature Warning

It is always a good idea to learn from your own mistakes. It is usually a 
better 
idea to learn from others’ mistakes, so you do not have to make them yourself. 
from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'


EARTH has a Right To Life,
otherwise we all die.



- Original Message 
From: Estrada Groups 
To: "solr-user@lucene.apache.org" 
Sent: Tue, January 4, 2011 10:33:29 AM
Subject: Re: Sub query using SOLR?

I am +1 on the interest on how to do this!

Adam


On Jan 4, 2011, at 1:26 PM, bbarani  wrote:

> 
> Hi,
> 
> I am trying to use subquery in SOLR, is there a way to implement this using
> SOLR query syntax?
> 
> Something like
> 
> Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)
> 
> The thing I really want is to use output of one query to be the input of
> another query. 
> 
> Not sure if it is possible to use the query() function (function query) for
> my case..
> 
> Just want to know if ther is a better approach...
> 
> Thanks,
> Barani
> -- 
> View this message in context: 
>http://lucene.472066.n3.nabble.com/Sub-query-using-SOLR-tp2193251p2193251.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Sub query using SOLR?

[dan whelan]

Something like this might work. It's two dismax queries joined by an AND

q=_query_:"{!dismax mm='100%25' qf='query fields and boosts for one 
query' v=$q1}" AND _query_:"{!dismax mm='100%25' qf='query fields and 
boosts for second query' v=$q2}"&q1=query one&q2=query two


Just change the qf params in both queries to the fields you are 
searching on and change q1 and q2 to the values being searched for.





On 1/4/11 10:26 AM, bbarani wrote:

Hi,

I am trying to use subquery in SOLR, is there a way to implement this using
SOLR query syntax?

Something like

Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)

The thing I really want is to use output of one query to be the input of
another query.

Not sure if it is possible to use the query() function (function query) for
my case..

Just want to know if ther is a better approach...

Thanks,
Barani

Re: Sub query using SOLR?

[Estrada Groups]

I am +1 on the interest on how to do this!

Adam


On Jan 4, 2011, at 1:26 PM, bbarani  wrote:

> 
> Hi,
> 
> I am trying to use subquery in SOLR, is there a way to implement this using
> SOLR query syntax?
> 
> Something like
> 
> Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)
> 
> The thing I really want is to use output of one query to be the input of
> another query. 
> 
> Not sure if it is possible to use the query() function (function query) for
> my case..
> 
> Just want to know if ther is a better approach...
> 
> Thanks,
> Barani
> -- 
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Sub-query-using-SOLR-tp2193251p2193251.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Sub query using SOLR?

[bbarani]

Hi,

I am trying to use subquery in SOLR, is there a way to implement this using
SOLR query syntax?

Something like

Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)

The thing I really want is to use output of one query to be the input of
another query. 

Not sure if it is possible to use the query() function (function query) for
my case..

Just want to know if ther is a better approach...

Thanks,
Barani
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Sub-query-using-SOLR-tp2193251p2193251.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Replication: the web application [/solr] .. likely to create a memory leak

[Markus Jelsma]

Is it possible this problem has something to do with my old index files not 
being removed? This problem only surfaces in my setup when i restart with 
replication on the slave. I can confirm that for some reason my replicated 
indexes get messed up only when i start restarting Tomcat several times.

On Tuesday 04 January 2011 15:48:31 Yonik Seeley wrote:
> On Tue, Jan 4, 2011 at 9:34 AM, Robert Muir  wrote:
> >[junit] WARNING: test class left thread running:
> > Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
> 
> I suppose we should move MultiThreadedHttpConnectionManager to
> CoreContainer.
> 
> -Yonik
> http://www.lucidimagination.com

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: searching against unstemmed text

[Jonathan Rochkind]

Do you have to do anything special to search against a field in Solr?  
No, that's what Solr does.


Please be more specific about what you are trying to do, what you expect 
to happen, and what happens instead.


If your Solr field is analyzed to stem, then indeed you can only match 
stemmed tokens, because that's the only tokens that are there.  You can 
create a different solr field that is not stemmed for wildcard searches 
if you like, which is perhaps what you're trying to do, but you haven't 
really told us.


On 1/4/2011 10:00 AM, Wodek Siebor wrote:

I'm trying to search using "text_rev" field, which is by default enabled in
the schema.xml,
but it doesn't work at all. Do I have to do anything special here.

I want to search using wildcards and searching against "text" field works
fine, except I can only find results against stemmed text.

Thanks,
Wlodek

Solr and UIMA #2

[Tommaso Teofili]

Hi all,
just a quick notice to let you know that a new component to consume UIMA
objects to a (local or remote) Solr instance is available inside UIMA
sandbox [1].
Note that this "writes" to Solr from UIMA pipelines (push) while in
SOLR-2129 [2] Solr "asks" UIMA to extract metadata while indexing
documents (pull) .
I'm going to write some documentation about Solr and UIMA on
http://wiki.apache.org/solr/SolrUIMA .
Cheers,
Tommaso

[1] : http://svn.apache.org/repos/asf/uima/sandbox/trunk/Solrcas/
[2] : https://issues.apache.org/jira/browse/SOLR-2129

Re: problems with replication

[Roxana Angheluta]

Thanks for replying.

I am not able to reproduce it, it does not happen at every replication.
I just see the errors in the logs.

At this point I have multiple errors in the logs and I am trying to analyze 
them systematically. See other examples below, this time from the slave. I 
suspect they are related and have to do with the replication. I used to have 
the server set-up as a repeater, now it is a simple slave.

I don't really know where to start debugging, any advice is appreciated.

roxana
 

Jan 4, 2011 5:06:22 AM sun.reflect.GeneratedMethodAccessor6 invoke
WARNING: Committed before 500 null

org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:767)
at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:562)
at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:898)
at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:528)
at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:619)
at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:573)
at 
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:183)
at 
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89)
at 
org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:45)
at 
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:322)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)

-

java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:972)
at org.mortbay.jetty.Response.sendError(Response.java:238)
at 
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at 
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)


--- On Mon, 1/3/11, Grant Ingersoll  wrote:

> From: Grant Ingersoll 
> Subject: Re: problems with replication
> To: solr-user@lucene.apache.org
> Date: Monday, January 3, 2011, 4:20 PM
> Are you able to reproduce this? 
> Can you give more details on what is happening?  Does
> this happen every time you replicate?
> 
> On Dec 29, 2010, at 3:11 PM, Roxana Angheluta wrote:
> 
> > Hi,
> > 
> > What can cause the following error on the master -
> stack trace below.
> > The slave is stuck in the middle of replication until
> manual intervention and responds very slow to searches.
> > 
> > Thanks,
> > roxana
> > 
> > Dec 29, 2010 8:25:52 PM
> org.apache.solr.handler.ReplicationHandler$FileStream write
> > WARNING: Exception while writing response for params:
> command=filecontent&checksum=true&indexversion=1291648857995&wt=filestream&file=_byb.fdt
> > org.mortbay.jetty.EofException
> >  

searching against unstemmed text

[Wodek Siebor]

I'm trying to search using "text_rev" field, which is by default enabled in
the schema.xml,
but it doesn't work at all. Do I have to do anything special here.

I want to search using wildcards and searching against "text" field works
fine, except I can only find results against stemmed text.

Thanks,
Wlodek
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/searching-against-unstemmed-text-tp2192026p2192026.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Replication: the web application [/solr] .. likely to create a memory leak

[Yonik Seeley]

On Tue, Jan 4, 2011 at 9:34 AM, Robert Muir  wrote:
>    [junit] WARNING: test class left thread running:
> Thread[MultiThreadedHttpConnectionManager cleanup,5,main]

I suppose we should move MultiThreadedHttpConnectionManager to CoreContainer.

-Yonik
http://www.lucidimagination.com

RE: Setting up Solr for PDFs on JBoss

[Olson, Ron]

This is what I have; I didn't alter it so I believe it's the default:


  

  
  text
  true
  ignored_

  
  true
  links
  ignored_

  

-Original Message-
From: Grant Ingersoll [mailto:gsing...@apache.org]
Sent: Monday, January 03, 2011 8:10 PM
To: solr-user@lucene.apache.org
Subject: Re: Setting up Solr for PDFs on JBoss

What's your solrconfig.xml look like for setting up the ExtractingReqHandler?

-Grant

On Jan 3, 2011, at 4:44 PM, Olson, Ron wrote:

> Hi all-
>
> After testing the PDF import functionality in my local copy of Solr 1.4.1 
> with the included Jetty app server, I tried replicating it using my copy of 
> Solr running in JBoss 5.10 (which uses Tomcat as its servlet container). When 
> I try to add a PDF, I get an error buried in the stack trace:
>
> Caused by: org.apache.solr.common.SolrException: Error Instantiating Request 
> Handler, org.apache.solr.handler.extraction.ExtractingRequestHandler is not a 
> org.apache.solr.request.SolrRequestHandler
>
>
> I am using multiple cores, but they all use the common "lib" directory, 
> instead of the "core/lib" directory. This lib directory is what is added to 
> the classpath when JBoss starts ($JBOSS_HOME/server/solr_test/lib), so all 
> the jars in this directory should be available to anything in the "deploy" 
> directory (just mentioning in case people aren't familiar with JBoss). I've 
> added all the jars from the contrib/extraction/lib directory, as well as the 
> jars from dist.
>
> My lib directory is effectively:
>
> apache-solr-cell-1.4.1.jareasymock.jar  
> lucene-spellchecker-2.9.3.jar
> apache-solr-clustering-1.4.1.jar  fontbox-0.1.0.jar 
> nekohtml-1.9.9.jar
> apache-solr-core-1.4.1.jargeronimo-stax-api_1.0_spec-1.0.1.jar  
> ojdbc14.jar
> apache-solr-solrj-1.4.1.jar   geronimo-stax-api_1.0_spec-1.0.jar
> ooxml-schemas-1.0.jar
> asm-3.1.jar   icu4j-3.8.jar 
> pdfbox-0.7.3.jar
> bcmail-jdk14-136.jar  jcl-over-slf4j-1.5.5.jar  
> poi-3.5-beta6.jar
> bcprov-jdk14-136.jar  jempbox-0.2.0.jar 
> poi-ooxml-3.5-beta6.jar
> commons-codec-1.3.jar junit-4.3.jar 
> poi-scratchpad-3.5-beta6.jar
> commons-compress-1.0.jar  log4j-1.2.14.jar  
> slf4j-api-1.5.5.jar
> commons-csv-1.0-SNAPSHOT-r609327.jar  lucene-analyzers-2.9.3.jar
> slf4j-jdk14-1.5.5.jar
> commons-fileupload-1.2.1.jar  lucene-core-2.9.3.jar 
> tika-core-0.4.jar
> commons-httpclient-3.1.jarlucene-highlighter-2.9.3.jar  
> tika-parsers-0.4.jar
> commons-io-1.4.jarlucene-memory-2.9.3.jar   
> wstx-asl-3.2.7.jar
> commons-lang-2.1.jar  lucene-misc-2.9.3.jar 
> xercesImpl-2.8.1.jar
> commons-logging-1.1.1.jar lucene-queries-2.9.3.jar  
> xml-apis-1.0.b2.jar
> dom4j-1.6.1.jar   lucene-snowball-2.9.3.jar 
> xmlbeans-2.3.0.jar
>
> I know several of these jars are already essentially present in JBoss (log4j, 
> for example), but I'm at a loss as to what to remove/add to get it to work. 
> Anyone have any ideas of configuring it under JBoss? The other cores are 
> database-based (thus the use of ojdbc14.jar), and they work fine.
>
> Thanks for any help,
>
> Ron
>
> DISCLAIMER: This electronic message, including any attachments, files or 
> documents, is intended only for the addressee and may contain CONFIDENTIAL, 
> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
> recipient, you are hereby notified that any use, disclosure, copying or 
> distribution of this message or any of the information included in or with it 
> is  unauthorized and strictly prohibited.  If you have received this message 
> in error, please notify the sender immediately by reply e-mail and 
> permanently delete and destroy this message and its attachments, along with 
> any copies thereof. This message does not create any contractual obligation 
> on behalf of the sender or Law Bulletin Publishing Company.
> Thank you.

--
Grant Ingersoll
http://www.lucidimagination.com



DISCLAIMER: This electronic message, including any attachments, files or 
documents, is intended only for the addressee and may contain CONFIDENTIAL, 
PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
recipient, you are hereby notified that any use, disclosure, copying or 
distribution of this message or any of the information included in or with it 
is  unauthorized and strictly prohibited.  If you have received this message in 
error, please notify the sender immediately by reply e-mail and permanently 
delete and destroy this message and its attachments, along with any copies 
ther

Re: Replication: the web application [/solr] .. likely to create a memory leak

[Robert Muir]

On Tue, Jan 4, 2011 at 9:23 AM, Markus Jelsma
 wrote:
> Hi,
>
> Anyone seen this before when stopping of restarting Solr 1.4.1 running as
> slave under Tomcat 6?
>
> SEVERE: The web application [/solr] appears to have started a thread named
> [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This
> is very likely to create a memory leak.
>
> It can also trigger multiple related exceptions errors:
>
>
> Jan 4, 2011 3:18:13 PM org.apache.catalina.loader.WebappClassLoader
> clearReferencesThreads
> SEVERE: The web application [/solr] appears to have started a thread named
> [pool-1-thread-1] but has failed to stop it. This is very likely to create a
> memory leak.

fyi, you can also see this by running its unit test:

[junit] Testsuite: org.apache.solr.handler.TestReplicationHandler
[junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 25.997 sec
[junit] - Standard Error -
[junit] WARNING: test class left thread running:
Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
[junit] WARNING: test class left thread running:
Thread[pool-44-thread-1,5,main]

Replication: the web application [/solr] .. likely to create a memory leak

[Markus Jelsma]

Hi,

Anyone seen this before when stopping of restarting Solr 1.4.1 running as 
slave under Tomcat 6?

SEVERE: The web application [/solr] appears to have started a thread named 
[MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This 
is very likely to create a memory leak.

It does _not_ happen when i set enable="false" in the slave part of the 
replication RH. I haven't tested it under Jetty because it can be reproduced 
by toggling replication only.

I think it somehow relates to my other issue where old index files are deleted. 
There is a connection between regularly restarting Tomcat and the problem 
showing up: http://www.mail-archive.com/solr-
u...@lucene.apache.org/msg45067.html


It can also trigger multiple related exceptions errors:


Jan 4, 2011 3:18:13 PM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named 
[pool-1-thread-1] but has failed to stop it. This is very likely to create a 
memory leak.
Jan 4, 2011 3:18:13 PM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named 
[pool-3-thread-1] but has failed to stop it. This is very likely to create a 
memory leak.
Jan 4, 2011 3:18:13 PM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named 
[MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This 
is very likely to create a memory leak.


Below is a relevant part of the log with only one error:


Jan 4, 2011 3:09:47 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Jan 4, 2011 3:09:47 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=null path=null 
params={sort=sort_tijd+desc&start=0&event=firstSearcher&q=*:*&fq=catlevel1:"Boeken"&rows=30}
 
hits=325104 status=0 QTime=31 
Jan 4, 2011 3:09:47 PM org.apache.solr.core.SolrCore close
INFO: []  CLOSING SolrCore org.apache.solr.core.solrc...@5c5ddd3
Jan 4, 2011 3:09:47 PM org.apache.solr.core.SolrCore closeSearcher
INFO: [] Closing main searcher on request.
Jan 4, 2011 3:09:47 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Jan 4, 2011 3:09:47 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing searc...@35a8d460 main

fieldValueCache{lookups=6,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=0,cumulative_lookups=6,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=3,cumulative_evictions=0,item_f_bijzonderheden={field=f_bijzonderheden,memSize=13062096,tindexSize=52,time=21,phase1=10,nTerms=4,bigTerms=0,termInstances=665,uses=1},item_f_autoaccessoires={field=f_autoaccessoires,memSize=14156450,tindexSize=46,time=562,phase1=541,nTerms=19,bigTerms=0,termInstances=1077665,uses=1},item_f_sp_eigenschappen={field=f_sp_eigenschappen,memSize=13062188,tindexSize=46,time=186,phase1=125,nTerms=3,bigTerms=1,termInstances=436,uses=1}}

filterCache{lookups=130,hits=125,hitratio=0.96,inserts=183,evictions=0,size=157,warmupTime=0,cumulative_lookups=130,cumulative_hits=125,cumulative_hitratio=0.96,cumulative_inserts=183,cumulative_evictions=0}

queryResultCache{lookups=308,hits=124,hitratio=0.40,inserts=184,evictions=0,size=157,warmupTime=0,cumulative_lookups=308,cumulative_hits=124,cumulative_hitratio=0.40,cumulative_inserts=184,cumulative_evictions=0}

documentCache{lookups=4046,hits=2096,hitratio=0.51,inserts=1950,evictions=0,size=1933,warmupTime=0,cumulative_lookups=4046,cumulative_hits=2096,cumulative_hitratio=0.51,cumulative_inserts=1950,cumulative_evictions=0}
Jan 4, 2011 3:09:47 PM org.apache.solr.update.DirectUpdateHandler2 close
INFO: closing 
DirectUpdateHandler2{commits=0,autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
Jan 4, 2011 3:09:47 PM org.apache.solr.update.DirectUpdateHandler2 close
INFO: closed 
DirectUpdateHandler2{commits=0,autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
Jan 4, 2011 3:09:47 PM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads
SEVERE: The web application [/solr] appears to have started a thread named 
[MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This 
is very likely to create a memory leak.
Jan 4, 2011 3:09:48 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080


-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: DataImportHanlder - Multiple entities will step into each other

[yu shen]

Thanks for the prompt reply. Let me try. Delete is not a big deal for the
moment.

2011/1/4 Matti Oinas 

> I managed to do that by using TemplateTransformer
>
> 
>  
> 
> ...
>  
> 
> ...
> 
>
> Only problem is that delta import fails to perform delete to the
> index. It seems that TemplateTransformer is not used when performing
> delete so delete by id doesn't work.
>
>
>
> 2011/1/4 yu shen :
> > Hi All,
> >
> > I have a dataimporthandler config file as below. It contains multiple
> > entities:
> > 
> > >
> >
> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
> > />
> >
> >
> >
> >
> >
> > 
> >
> > All data are from a database. Problem is item/company and other entity
> all
> > have the field 'id', with value start from 1 to n. In this case,
> > item/company etc. will step into each other.
> > Is there a way to prevent is from happening. Such as designate different
> > entity to different partition.
> >
> > One way I can think of is to seperate different entity to different
> > instance, which is not ideal solution IMO.
> >
> > Would some one point me to a reference? And also give some instructions?
> >
>

Re: DataImportHanlder - Multiple entities will step into each other

[Matti Oinas]

I managed to do that by using TemplateTransformer


  
 
...
  
 
...


Only problem is that delta import fails to perform delete to the
index. It seems that TemplateTransformer is not used when performing
delete so delete by id doesn't work.



2011/1/4 yu shen :
> Hi All,
>
> I have a dataimporthandler config file as below. It contains multiple
> entities:
> 
>        
> url="jdbc:mysql://localhost:1521/changan?useUnicode=true&characterEncoding=utf8&autoReconnect=true"...
> />
>        
>                
>                
>                
>        
> 
>
> All data are from a database. Problem is item/company and other entity all
> have the field 'id', with value start from 1 to n. In this case,
> item/company etc. will step into each other.
> Is there a way to prevent is from happening. Such as designate different
> entity to different partition.
>
> One way I can think of is to seperate different entity to different
> instance, which is not ideal solution IMO.
>
> Would some one point me to a reference? And also give some instructions?
>

DataImportHanlder - Multiple entities will step into each other

[yu shen]

Hi All,

I have a dataimporthandler config file as below. It contains multiple
entities:









All data are from a database. Problem is item/company and other entity all
have the field 'id', with value start from 1 to n. In this case,
item/company etc. will step into each other.
Is there a way to prevent is from happening. Such as designate different
entity to different partition.

One way I can think of is to seperate different entity to different
instance, which is not ideal solution IMO.

Would some one point me to a reference? And also give some instructions?

Re: Does Solr/Lucene support Localization/Internationalization

[prasad deshpande]

My Primary interest is in Solr.

Thanks,
Prasad

On Tue, Jan 4, 2011 at 6:25 PM, Erick Erickson wrote:

> First, it's no big deal, but it's better to just post to one list only.
> I'm not sure whether your primary interest is in Solr or Lucene
>
> See: http://lucene.apache.org/java/3_0_3/api/all/index.html
> for a list of
> Lucene
> Analyzers for various languages.
>
> See:  http://wiki.apache.org/solr/LanguageAnalysis
> for Solr language support
>
> Best
> Erick
>
> On Tue, Jan 4, 2011 at 7:43 AM, prasad deshpande <
> prasad.deshpand...@gmail.com> wrote:
>
> > >
> > > Hello,
> > >
> > > I am completely new for Solr/lucene. I am going through documents
> > provided
> > > by Apache.
> > > Can any one tell me that Lucene/Solr supports indexing and searching
> for
> > > other than English language?
> > >
> > > It seems Solr/Lucene is completely developed in Java.
> > > I want to integrate Solr/Lucene with C/CPP client. Is it possible? If
> > yes,
> > > how?
> > >
> > > Thanks and Regards,
> > > Prasad
> > >
> >
>

Re: Does Solr/Lucene support Localization/Internationalization

[Erick Erickson]

First, it's no big deal, but it's better to just post to one list only.
I'm not sure whether your primary interest is in Solr or Lucene

See: http://lucene.apache.org/java/3_0_3/api/all/index.html
for a list of Lucene
Analyzers for various languages.

See:  http://wiki.apache.org/solr/LanguageAnalysis
for Solr language support

Best
Erick

On Tue, Jan 4, 2011 at 7:43 AM, prasad deshpande <
prasad.deshpand...@gmail.com> wrote:

> >
> > Hello,
> >
> > I am completely new for Solr/lucene. I am going through documents
> provided
> > by Apache.
> > Can any one tell me that Lucene/Solr supports indexing and searching for
> > other than English language?
> >
> > It seems Solr/Lucene is completely developed in Java.
> > I want to integrate Solr/Lucene with C/CPP client. Is it possible? If
> yes,
> > how?
> >
> > Thanks and Regards,
> > Prasad
> >
>

Does Solr/Lucene support Localization/Internationalization

[prasad deshpande]

>
> Hello,
>
> I am completely new for Solr/lucene. I am going through documents provided
> by Apache.
> Can any one tell me that Lucene/Solr supports indexing and searching for
> other than English language?
>
> It seems Solr/Lucene is completely developed in Java.
> I want to integrate Solr/Lucene with C/CPP client. Is it possible? If yes,
> how?
>
> Thanks and Regards,
> Prasad
>

Re: Setting up Solr for PDFs on JBoss

[Jak Akdemir]

In JBoss, duplicate libraries will be ignored as you mentioned. You may
start to find libraries used in JBoss with "find -name *.jar". I don't know
any other resource than wiki. It says remove the libraries below.

   - xercesImpl-2.8.1.jar
   - xml-apis-1.3.03.jar

 http://wiki.apache.org/solr/SolrJBoss


On Mon, Jan 3, 2011 at 11:44 PM, Olson, Ron  wrote:

> Hi all-
>
> After testing the PDF import functionality in my local copy of Solr 1.4.1
> with the included Jetty app server, I tried replicating it using my copy of
> Solr running in JBoss 5.10 (which uses Tomcat as its servlet container).
> When I try to add a PDF, I get an error buried in the stack trace:
>
> Caused by: org.apache.solr.common.SolrException: Error Instantiating
> Request Handler, org.apache.solr.handler.extraction.ExtractingRequestHandler
> is not a org.apache.solr.request.SolrRequestHandler
>
>
> I am using multiple cores, but they all use the common "lib" directory,
> instead of the "core/lib" directory. This lib directory is what is added to
> the classpath when JBoss starts ($JBOSS_HOME/server/solr_test/lib), so all
> the jars in this directory should be available to anything in the "deploy"
> directory (just mentioning in case people aren't familiar with JBoss). I've
> added all the jars from the contrib/extraction/lib directory, as well as the
> jars from dist.
>
> My lib directory is effectively:
>
> apache-solr-cell-1.4.1.jareasymock.jar
>  lucene-spellchecker-2.9.3.jar
> apache-solr-clustering-1.4.1.jar  fontbox-0.1.0.jar
> nekohtml-1.9.9.jar
> apache-solr-core-1.4.1.jargeronimo-stax-api_1.0_spec-1.0.1.jar
>  ojdbc14.jar
> apache-solr-solrj-1.4.1.jar   geronimo-stax-api_1.0_spec-1.0.jar
>  ooxml-schemas-1.0.jar
> asm-3.1.jar   icu4j-3.8.jar
> pdfbox-0.7.3.jar
> bcmail-jdk14-136.jar  jcl-over-slf4j-1.5.5.jar
>  poi-3.5-beta6.jar
> bcprov-jdk14-136.jar  jempbox-0.2.0.jar
> poi-ooxml-3.5-beta6.jar
> commons-codec-1.3.jar junit-4.3.jar
> poi-scratchpad-3.5-beta6.jar
> commons-compress-1.0.jar  log4j-1.2.14.jar
>  slf4j-api-1.5.5.jar
> commons-csv-1.0-SNAPSHOT-r609327.jar  lucene-analyzers-2.9.3.jar
>  slf4j-jdk14-1.5.5.jar
> commons-fileupload-1.2.1.jar  lucene-core-2.9.3.jar
> tika-core-0.4.jar
> commons-httpclient-3.1.jarlucene-highlighter-2.9.3.jar
>  tika-parsers-0.4.jar
> commons-io-1.4.jarlucene-memory-2.9.3.jar
> wstx-asl-3.2.7.jar
> commons-lang-2.1.jar  lucene-misc-2.9.3.jar
> xercesImpl-2.8.1.jar
> commons-logging-1.1.1.jar lucene-queries-2.9.3.jar
>  xml-apis-1.0.b2.jar
> dom4j-1.6.1.jar   lucene-snowball-2.9.3.jar
> xmlbeans-2.3.0.jar
>
> I know several of these jars are already essentially present in JBoss
> (log4j, for example), but I'm at a loss as to what to remove/add to get it
> to work. Anyone have any ideas of configuring it under JBoss? The other
> cores are database-based (thus the use of ojdbc14.jar), and they work fine.
>
> Thanks for any help,
>
> Ron
>
> DISCLAIMER: This electronic message, including any attachments, files or
> documents, is intended only for the addressee and may contain CONFIDENTIAL,
> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended
> recipient, you are hereby notified that any use, disclosure, copying or
> distribution of this message or any of the information included in or with
> it is  unauthorized and strictly prohibited.  If you have received this
> message in error, please notify the sender immediately by reply e-mail and
> permanently delete and destroy this message and its attachments, along with
> any copies thereof. This message does not create any contractual obligation
> on behalf of the sender or Law Bulletin Publishing Company.
> Thank you.
>

Re: use query to set filter on fields

[Raymi]

I'm not sure, whether I made myself very clear. I don't
think that it is difficult to evaluate the query. Here's a
(very inefficient) algorithm in pseudocode that would do,
what I have in mind:

query = ...

foreach (term in query)
   foreach (field in SOLR.config.fields)
 result = SOLR.query(field:term)
 if result.size>  0
   filters.addFilter(field=term)
   break
 end
   end
end

return SOLR.filter(filters)


Okey if you have already something, I think simplest place to put this is a 
custom search handler. http://wiki.apache.org/solr/SolrPlugins
Other option can be QParserPlugin.


public final class CustomSearchHandler extends SearchHandler {

@Override
   public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) 
throws IOException, ParseException, InstantiationException, 
IllegalAccessException {


   ModifiableSolrParams solrParams = new   
ModifiableSolrParams(req.getParams());
   String query = req.getParams().get(CommonParams.Q, "*:*");

foreach (term in query)
foreach (field in SOLR.config.fields)  =>  
http://wiki.apache.org/solr/LukeRequestHandler
  result = SOLR.query(field:term)
  if result.size>  0
filters.addFilter(field=term)  =>  solrParams.add(CommonParams.FQ, 
"field:term");
break
  end
end
  end

  req.setParams(solrParams);
  super.handleRequestBody(req, rsp);
}


Ok. Thank you very much for your help and the starting points. I'll give 
it a try. Maybe I come up with a smarter way than my "brute force" 
algorithm.

Re: use query to set filter on fields

[Ahmet Arslan]

> I'm not sure, whether I made myself very clear. I don't
> think that it is difficult to evaluate the query. Here's a
> (very inefficient) algorithm in pseudocode that would do,
> what I have in mind:
> 
> query = ...
> 
> foreach (term in query)
>   foreach (field in SOLR.config.fields)
>     result = SOLR.query(field:term)
>     if result.size > 0
>       filters.addFilter(field=term)
>       break
>     end
>   end
> end
> 
> return SOLR.filter(filters)

Okey if you have already something, I think simplest place to put this is a 
custom search handler. http://wiki.apache.org/solr/SolrPlugins
Other option can be QParserPlugin.


public final class CustomSearchHandler extends SearchHandler {

@Override
  public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) 
throws IOException, ParseException, InstantiationException, 
IllegalAccessException {

   
  ModifiableSolrParams solrParams = new   ModifiableSolrParams(req.getParams());
  String query = req.getParams().get(CommonParams.Q, "*:*");

   foreach (term in query)
   foreach (field in SOLR.config.fields)  => 
http://wiki.apache.org/solr/LukeRequestHandler
 result = SOLR.query(field:term)
 if result.size > 0
   filters.addFilter(field=term)  => solrParams.add(CommonParams.FQ, 
"field:term");
   break
 end
   end
 end
 
 req.setParams(solrParams);
 super.handleRequestBody(req, rsp);
} 

Re: use query to set filter on fields

[Raymi]

No, I don't.
But since I'll store documents in solr having "color" set
to "red",
"green", "blue" and "manufacturer" set to "ford", "bmw"
etc. the
information is available.


Okey information is available but the difficult part is recognize those things 
from the free form text query. There is no such mechanism is Solr.

If you can build such program, you can inject it to solr using a custom solr 
plugin.



I'm not sure, whether I made myself very clear. I don't think that it is 
difficult to evaluate the query. Here's a (very inefficient) algorithm 
in pseudocode that would do, what I have in mind:


query = ...

foreach (term in query)
  foreach (field in SOLR.config.fields)
result = SOLR.query(field:term)
if result.size > 0
  filters.addFilter(field=term)
  break
end
  end
end

return SOLR.filter(filters)

Re: use query to set filter on fields

[Ahmet Arslan]

> No, I don't.
> But since I'll store documents in solr having "color" set
> to "red", 
> "green", "blue" and "manufacturer" set to "ford", "bmw"
> etc. the 
> information is available.

Okey information is available but the difficult part is recognize those things 
from the free form text query. There is no such mechanism is Solr.

If you can build such program, you can inject it to solr using a custom solr 
plugin.


  

Re: use query to set filter on fields

[Raymi]

I'm evaluating SOLR and have a question about query
processing. Is it possible to set filters on fields
according to the terms entered in the query? I know that I
can use dismax to search in multiple fields. However, what I
want to achieve for a given query is the following:
  - parse all terms in the query
  - determine values of different fields that match these
terms
  - set filters for these values
  - return all documents matching the filters

Example:
documents representing cars with fields "manufacturer" and
"color"
query for "red ford" sets filters manufacturer=ford,
color=red

One can then use faceted navigation to further drill down.


So do you have an existing java program that recognizes red as color and ford 
as manufacturer? And


No, I don't.
But since I'll store documents in solr having "color" set to "red", 
"green", "blue" and "manufacturer" set to "ford", "bmw" etc. the 
information is available.

Re: use query to set filter on fields

[Ahmet Arslan]

> I'm evaluating SOLR and have a question about query
> processing. Is it possible to set filters on fields
> according to the terms entered in the query? I know that I
> can use dismax to search in multiple fields. However, what I
> want to achieve for a given query is the following:
>  - parse all terms in the query
>  - determine values of different fields that match these
> terms
>  - set filters for these values
>  - return all documents matching the filters
> 
> Example:
> documents representing cars with fields "manufacturer" and
> "color"
> query for "red ford" sets filters manufacturer=ford,
> color=red
> 
> One can then use faceted navigation to further drill down.

So do you have an existing java program that recognizes red as color and ford 
as manufacturer? And 


  

use query to set filter on fields

[Raymi]

Hi,

I'm evaluating SOLR and have a question about query processing. Is it 
possible to set filters on fields according to the terms entered in the 
query? I know that I can use dismax to search in multiple fields. 
However, what I want to achieve for a given query is the following:

 - parse all terms in the query
 - determine values of different fields that match these terms
 - set filters for these values
 - return all documents matching the filters

Example:
documents representing cars with fields "manufacturer" and "color"
query for "red ford" sets filters manufacturer=ford, color=red

One can then use faceted navigation to further drill down.

Is this possible with SOLR?

Thanks.
Cheers,

Dominik

Re: Must require quote with single word token query?

[Chamnap Chhorn]

Very late reply, actually.

I can't manage it to work with local params using my text_keyword field and
multi word query. What i do to achieve is to do full text searching. If the
query matches the keyphrase, it will be in higher position.

Are they other ways to work around?

On Wed, Nov 24, 2010 at 9:13 PM, Jonathan Rochkind  wrote:

> Okay, we need to take a step back and think about what you are trying to
> do.
>
> Reading back in the thread and looking at your schema, you have a
> non-tokenized field whose terms can include whitespace.  There is in fact no
> good way to use that with dismax, dismax doesn't work that way. What you can
> do as Yonik suggests is use the 'field' query parser instead.  You can force
> the use of the 'field' query parser with 'local params', or you can even
> create a combined query with uses 'field' for one clause and 'dismax' for
> another, with nested query syntax.
>
> But every individual part of your query can only use one query parser at a
> time, there's no way to use both at once.
>
> But if you want to use that request handler, but force it to use 'field'
> _instead_, that can be easily done:
>
> &q=multi word query&defType=field&v=field_name
>
> You don't even need 'local params', although you can also do it with 'local
> params': &q={!field v=field_name}
>
> That's it. (except all those values need to be URI encoded).  But it won't
> be using dismax anymore, although it'll be using that request handler you
> have set up to default to dismax, you're telling it to use 'field' this time
> anyway.
>
> If that doesn't do what you want, why don't you take a step back and tell
> us what query behavior you are actually trying to create, and maybe someone
> can give you some ideas for accomplishing it.
>
> 
> From: Chamnap Chhorn [chamnapchh...@gmail.com]
> Sent: Wednesday, November 24, 2010 4:43 AM
> To: yo...@lucidimagination.com
> Cc: solr-user@lucene.apache.org
> Subject: Re: Must require quote with single word token query?
>
> I've looked at solr local params. However, I can't figure out how to
> integrate it with my full text search using dismax handler. Here is my full
> text search request handler.
>
>  
>
>  explicit
>  20
>  dismax
>  name_ngram^20 name^40 postal_code address description
> long_description location keyphrase short_description category telephone
> email website
>  name_ngram
>  fap^10
>  uuid
>  2.2
>  on
>  0.1
>
>
>  type:Listing
>
>
>  false
>
>
>  spellcheck
>  elevateListings
>
>  
>
> Note: postal_code, keyphrase, category, telephone, email, website has field
> type "text_keyword".
>
> Thanks
> On Sat, Nov 20, 2010 at 9:49 AM, Yonik Seeley  >wrote:
>
> > On Fri, Nov 19, 2010 at 9:41 PM, Chamnap Chhorn  >
> > wrote:
> > > Wow, i never know this syntax before. What's that called?
> >
> > I dubbed it "local params" since it adds local info to a parameter
> > (think extra metadata, like XML attributes on an element).
> >
> > http://wiki.apache.org/solr/LocalParams
> >
> > It's used mostly to invoke different query parsers, but it's also used
> > to add extra metadata to faceting commands too (and is required for
> > stuff like multi-select faceting):
> >
> >
> >
> http://wiki.apache.org/solr/SimpleFacetParameters#Multi-Select_Faceting_and_LocalParams
> >
> >
> > -Yonik
> > http://www.lucidimagination.com
> >
> >
> >
> > > On 11/19/10, Yonik Seeley  wrote:
> > >> On Tue, Nov 16, 2010 at 10:28 PM, Chamnap Chhorn
> > >>  wrote:
> > >>> I have one question related to single word token with dismax query.
> In
> > >>> order
> > >>> to be found I need to add the quote around the search query all the
> > time.
> > >>> This is quite hard for me to do since it is part of full text search.
> > >>>
> > >>> Here is my solr query and field type definition (Solr 1.4):
> > >>> > >>> positionIncrementGap="100">
> > >>>  
> > >>>
> > >>>
> > >>>
> > >>> > >>> words="stopwords.txt" enablePositionIncrements="true"/>
> > >>> > synonyms="synonyms.txt"
> > >>> ignoreCase="true" expand="false" />
> > >>>
> > >>>  
> > >>>
> > >>>
> > >>> > >>> stored="false" multiValued="true"/>
> > >>>
> > >>> With this query
> > >>> q=smart%20mobile&qf=keyphrase&debugQuery=on&defType=dismax,
> > >>> solr returns nothing. However, with quote on the search query
> q="smart
> > >>> mobile"&qf=keyphrase&debugQuery=on&defType=dismax, the result is
> found.
> > >>>
> > >>> Is it a must to use quote for a single word token field?
> > >>
> > >> Yes, you must currently quote tokens if they contain whitespace -
> > >> otherwise the query parser first breaks on whitespace before doing
> > >> analysis on each part separately.
> > >>
> > >> Using dismax is an odd choice if you are only querying on keyphrase
> > though.
> > >> You might look at the field query parser - it is a basic single-field
> > >> s