Re: Document Level Security (SOLR-1872 ,SOLR,SOLR-1834)

[Constantijn Visinescu]

Just to chip in my 2 cents:

You know you can increase the max number of boolean clauses in the
configuration files?
Depending on your situation it might not be a permanent fix, but it
could provide some instant relief.

Constantijn


On Fri, Jun 17, 2011 at 11:19 AM, Peter Sturge  wrote:
> You'll need to be a bit careful using joins, as the performance hit
> can be significant if you have lots of cross-referencing to do, which
> I believe you would given your scenario.
>
> Your table could be setup to use the username as the key (for fast
> lookup), then map these to your own data class or collection or
> similar to hold your other information: products, expiry etc.
> By using your own data class, it's then easy to extend it later if you
> want to add additional parameters. (for example: HashMap MyDataClass>)
>
> When a search comes in, the user is looked up to retrieve the data
> class, then its contents (as defined by you) is examined and the query
> is processed/filtered appropriately.
>
> You'll need a bootstrap mechanism for populating the list in the first
> place. One thing worth looking at is lazy loading - i.e. the first
> time a user does a search (you lookup the user in the table, and it
> isn't there), you load the data class (maybe from your DB, a file, or
> index), then ad it to the table. This is good if you have 10's of
> thousands or millions of users, but only a handful are actually
> searching, some perhaps very rarely.
>
> If you do have millions of users, and your data class has heavy
> requirements (e.g. many thousands of products + info etc.), you might
> want to 'time-out' in-memory table entries, if the table gets really
> huge - it depends on the usage of your system. (you can run a
> synchronized cleanup thread to do this if you deemed it necessary).
>
>
> On Fri, Jun 17, 2011 at 6:06 AM, Sujatha Arun  wrote:
>> Alexey,
>>
>> Do you mean that we  have current Index as it is and have a separate core
>> which  has only the user-id ,product-id relation and at while querying ,do a
>> join between the two cores based on the user-id.
>>
>>
>> This would involve us to Index/delete the product  as and when the user
>> subscription for a product changes ,This would involve some amount of
>> latency if the Indexing (we have a queue system for Indexing across the
>> various instances) or deletion is delayed
>>
>> IF we want to go ahead with this solution ,We currently are using solr 1.3
>> , so  is this functionality available as a patch for solr 1.3?Would it be
>> possible to  do with a separate Index  instead of a core ,then I can create
>> only one  Index common for all our instances and then use this instance to
>> do the join.
>>
>> Thanks
>> Sujatha
>>
>> On Thu, Jun 16, 2011 at 9:27 PM, Alexey Serba  wrote:
>>
>>> > So a search for a product once the user logs in and searches for only the
>>> > products that he has access to Will translate to something like this .
>>> ,the
>>> > product ids are obtained form the db  for a particular user and can run
>>> > into  n  number.
>>> >
>>> >  &fq=product_id(100 10001  ..n number)
>>> >
>>> > but we are currently running into too many Boolean expansion error .We
>>> are
>>> > not able to tie the user also into roles as each user is mainly any one
>>> who
>>> > comes to site and purchases a product .
>>>
>>> I'm wondering if new trunk Solr join functionality can help here.
>>>
>>> * http://wiki.apache.org/solr/Join
>>>
>>> In theory you can index your products (product_id, ...) and
>>> user_id-product many-to-many relation (user_product_id, user_id) into
>>> signle/different cores and then do join, like
>>> f=search terms&fq={!join from=product_id to=user_product_id}user_id:10101
>>>
>>> But I haven't tried that, so I'm just speculating.
>>>
>>
>

Re: Getting and viewing a heap dump

[Constantijn Visinescu]

I was sending using the gmail webbrowser client in plaintext.

Spamassasin didn't seem to like 3 things (according to the error
message i got back):
- I use a free email adress
- My email address (before the @) ends with a number
- The email had the word replication in the subject line.

No idea where rule #3 came from but it was the easiest to fix so
that's what I changed ;)


On Tue, May 31, 2011 at 4:21 PM, Erick Erickson  wrote:
> Constantjin:
>
> I've had better luck by sending messages as "plain text". The Spam
> filter on the user
> list sometimes acts up if you send mail in "richtext" or similar
> formats. Gmail has a link
> to change this, what client are you using?
>
> And thanks for participating!
>
> Best
> Erick
>
> On Tue, May 31, 2011 at 3:22 AM, Constantijn Visinescu
>  wrote:
>> Hi Bernd,
>>
>> I'm assuming Linux here, if you're running something else these
>> instructions might differ slightly.
>>
>> First get a heap dump with:
>> jmap -heap:format=b,file=/path/to/generate/heapdumpfile.hprof 1234
>>
>> with 1234 being the PID (process id) of the JVM
>>
>> After you get a Heap dump you can analyze it with Eclipse MAT (Memory
>> Analyzer Tool).
>>
>> Just a heads up if you're doing this in production: the JVM will
>> freeze completely while generating the heap dump, which will seem like
>> a giant stop the world GC with a 10GB heap.
>>
>> Good luck with finding out what's eating your memory!
>>
>> Constantijn
>>
>> P.S.
>> Sorry about  altering the subject line, but the spam assassin used by
>> the mailing list was rejecting my post because it had replication in
>> the subject line. hope it doesn't mess up the thread.
>>
>> On Tue, May 31, 2011 at 8:43 AM, Bernd Fehling
>>  wrote:
>>> Some more info,
>>> after one week the servers have the following status:
>>>
>>> Master (indexing only)
>>> + looks good and has heap size of about 6g from 10g OldGen
>>> + has loaded meanwhile 2 times the index from scratch via DIH
>>> + has added new documents into existing index via DIH
>>> + has optimized and replicated
>>> + no full GC within one week
>>>
>>> Slave A (search only) Online
>>> - looks bad and has heap size of 9.5g from 10g OldGen
>>> + was replicated
>>> - several full GC
>>>
>>> Slave B (search only) Backup
>>> + looks good has heap size of 4 g from 10g OldGen
>>> + was replicated
>>> + no full GC within one week
>>>
>>> Conclusion:
>>> + DIH, processing, indexing, replication are fine
>>> - the search is crap and "eats up" OldGen heap which can't be
>>>  cleaned up by full GC. May be memory leaks or what ever...
>>>
>>> Due to this Solr 3.1 can _NOT_ be recommended as high-availability,
>>> high-search-load search engine because of unclear heap problems
>>> caused by the search. The search is "out of the box", so no
>>> self produced programming errors.
>>>
>>> Any tools available for JAVA to analyze this?
>>> (like valgrind or electric fence for C++)
>>>
>>> Is it possible to analyze a heap dump produced with jvisualvm?
>>> Which tools?
>>>
>>>
>>> Bernd
>>>
>>>
>>> Am 30.05.2011 15:51, schrieb Bernd Fehling:
>>>>
>>>> Dear list,
>>>> after switching from FAST to Solr I get the first _real_ data.
>>>> This includes search times, memory consumption, perfomance of solr,...
>>>>
>>>> What I recognized so far is that something eats up my OldGen and
>>>> I assume it might be replication.
>>>>
>>>> Current Data:
>>>> one master - indexing only
>>>> two slaves - search only
>>>> over 28 million docs
>>>> single instance
>>>> single core
>>>> index size 140g
>>>> current heap size 16g
>>>>
>>>> After startup I have about 4g heap in use and about 3.5g of OldGen.
>>>> After one week and some replications OldGen is filled close to 100
>>>> percent.
>>>> If I start an optimize under this condition I get OOM of heap.
>>>> So my assumption is that something is eating up my heap.
>>>>
>>>> Any idea how to trace this down?
>>>>
>>>> May be a memory leak somewhere?
>>>>
>>>> Best regards
>>>> Bernd
>>>>
>>>
>>
>

Getting and viewing a heap dump

[Constantijn Visinescu]

Hi Bernd,

I'm assuming Linux here, if you're running something else these
instructions might differ slightly.

First get a heap dump with:
jmap -heap:format=b,file=/path/to/generate/heapdumpfile.hprof 1234

with 1234 being the PID (process id) of the JVM

After you get a Heap dump you can analyze it with Eclipse MAT (Memory
Analyzer Tool).

Just a heads up if you're doing this in production: the JVM will
freeze completely while generating the heap dump, which will seem like
a giant stop the world GC with a 10GB heap.

Good luck with finding out what's eating your memory!

Constantijn

P.S.
Sorry about  altering the subject line, but the spam assassin used by
the mailing list was rejecting my post because it had replication in
the subject line. hope it doesn't mess up the thread.

On Tue, May 31, 2011 at 8:43 AM, Bernd Fehling
 wrote:
> Some more info,
> after one week the servers have the following status:
>
> Master (indexing only)
> + looks good and has heap size of about 6g from 10g OldGen
> + has loaded meanwhile 2 times the index from scratch via DIH
> + has added new documents into existing index via DIH
> + has optimized and replicated
> + no full GC within one week
>
> Slave A (search only) Online
> - looks bad and has heap size of 9.5g from 10g OldGen
> + was replicated
> - several full GC
>
> Slave B (search only) Backup
> + looks good has heap size of 4 g from 10g OldGen
> + was replicated
> + no full GC within one week
>
> Conclusion:
> + DIH, processing, indexing, replication are fine
> - the search is crap and "eats up" OldGen heap which can't be
>  cleaned up by full GC. May be memory leaks or what ever...
>
> Due to this Solr 3.1 can _NOT_ be recommended as high-availability,
> high-search-load search engine because of unclear heap problems
> caused by the search. The search is "out of the box", so no
> self produced programming errors.
>
> Any tools available for JAVA to analyze this?
> (like valgrind or electric fence for C++)
>
> Is it possible to analyze a heap dump produced with jvisualvm?
> Which tools?
>
>
> Bernd
>
>
> Am 30.05.2011 15:51, schrieb Bernd Fehling:
>>
>> Dear list,
>> after switching from FAST to Solr I get the first _real_ data.
>> This includes search times, memory consumption, perfomance of solr,...
>>
>> What I recognized so far is that something eats up my OldGen and
>> I assume it might be replication.
>>
>> Current Data:
>> one master - indexing only
>> two slaves - search only
>> over 28 million docs
>> single instance
>> single core
>> index size 140g
>> current heap size 16g
>>
>> After startup I have about 4g heap in use and about 3.5g of OldGen.
>> After one week and some replications OldGen is filled close to 100
>> percent.
>> If I start an optimize under this condition I get OOM of heap.
>> So my assumption is that something is eating up my heap.
>>
>> Any idea how to trace this down?
>>
>> May be a memory leak somewhere?
>>
>> Best regards
>> Bernd
>>
>

Re: Query speed decreased dramatically, not sure why though?

[Constantijn Visinescu]

What happens to your performance if you query for *:* instead of * ?
(probably have to url encode the colon)

On Tue, Aug 24, 2010 at 11:26 AM, C0re  wrote:
>
> We have a query which takes the form of
>
> ".../select?q=*&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> This query takes around 5 seconds to complete.
>
> I changed the query to the following;
>
> ".../select?q=[* TO NOW]&sort=evalDate+desc,score+desc&start=0&rows=10"
>
> The query now returns in around 600 milliseconds.
>
> Can any one help with explaining why [* TO NOW] has had such a big effect?
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Query-speed-decreased-dramatically-not-sure-why-though-tp1307636p1307636.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Problem in setting the request writer in SolrJ (wiki page wrong?)

[Constantijn Visinescu]

If my requests aren't serialized via a request writer then why does my
embedded solr crash when i comment out the following line in my
solrconfig:
  

it crashes with the exception that it can't with the /update URL. (I
left in the javabin request handler).




On Mon, Aug 23, 2010 at 10:40 PM, Ryan McKinley  wrote:
> Note that the 'setRequestWriter' is not part of the SolrServer API, it
> is on the CommonsHttpSolrServer:
> http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/CommonsHttpSolrServer.html#setRequestWriter%28org.apache.solr.client.solrj.request.RequestWriter%29
>
> If you are using EmbeddedSolrServer, the params are not serialized via
> RequestWriter, so you don't have any options there.
>
> ryan
>
>
> On Mon, Aug 23, 2010 at 9:24 AM, Constantijn Visinescu
>  wrote:
>> Hello,
>>
>> I'm using an embedded solrserver in my Java webapp, but as far as i
>> can tell it's defaulting to sending updates in XML, which seems like a
>> huge waste compared to sending it in Java binary format.
>>
>> According to this page:
>> http://wiki.apache.org/solr/Solrj#Setting_the_RequestWriter
>>
>> I'm supposed to be able to set the requestwriter like so:
>> server.setRequestWriter(new BinaryRequestWriter());
>>
>> However this method doesn't seem to exists in the SolrServer class of
>> SolrJ 1.4.1 ?
>>
>> How do i set it to process updates in the java binary format?
>>
>> Thanks in advance,
>> Constantijn Visinescu
>>
>> P.S.
>> I'm creating my SolrServer instance like this:
>>        private SolrServer solrServer;
>>        CoreContainer container = new 
>> CoreContainer.Initializer().initialize();
>>        solrServer = new EmbeddedSolrServer(container, "");
>>
>> this solrServer wont let me set a request writer.
>>
>

Problem in setting the request writer in SolrJ (wiki page wrong?)

[Constantijn Visinescu]

Hello,

I'm using an embedded solrserver in my Java webapp, but as far as i
can tell it's defaulting to sending updates in XML, which seems like a
huge waste compared to sending it in Java binary format.

According to this page:
http://wiki.apache.org/solr/Solrj#Setting_the_RequestWriter

I'm supposed to be able to set the requestwriter like so:
server.setRequestWriter(new BinaryRequestWriter());

However this method doesn't seem to exists in the SolrServer class of
SolrJ 1.4.1 ?

How do i set it to process updates in the java binary format?

Thanks in advance,
Constantijn Visinescu

P.S.
I'm creating my SolrServer instance like this:
private SolrServer solrServer;
CoreContainer container = new CoreContainer.Initializer().initialize();
solrServer = new EmbeddedSolrServer(container, "");

this solrServer wont let me set a request writer.

Re: Configuring multiple SOLR apps to play nice with MBeans / JMX

[Constantijn Visinescu]

> it would probably be pretty trivial to add if you want to take a stab at a
patch for it.
> -Hoss

*stab*
https://issues.apache.org/jira/browse/SOLR-1843
:)

Re: Configuring multiple SOLR apps to play nice with MBeans / JMX

[Constantijn Visinescu]

Don't know about other servlet containers, but i can confirm Resin 3 breaks
if you try to load 2 completely independent webapps into it that both use
solr with jmx enabled.

I also had a similar issue with Blaze DS (library for flash remoting that
I'm using to power the UI for my webapp), but Blaze DS uses the display-name
from the web.xml to register MBeans under. So making sure each webapp had a
different display name fixed that issue.

However solr always insists in using "solr" regardless of ... well pretty
much everything ;)

here's a part of the stack trace for informational purposes:

13:23:19,572  WARN JmxMonitoredMap:139 - Failed to register info bean:
org.apache.solr.highlight.HtmlFormatter
javax.management.InstanceNotFoundException:
solr:type=org.apache.solr.highlight.HtmlFormatter,id=org.apache.solr.highlight.HtmlFormatter
at
com.caucho.jmx.MBeanContext.unregisterMBean(MBeanContext.java:285)
at
com.caucho.jmx.AbstractMBeanServer.unregisterMBean(AbstractMBeanServer.java:477)
at
org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:135)
at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:47)
at
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:508)
at org.apache.solr.core.SolrCore.(SolrCore.java:605)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83)
at
com.caucho.server.dispatch.FilterManager.createFilter(FilterManager.java:144)
at
com.caucho.server.dispatch.FilterManager.init(FilterManager.java:91)
at com.caucho.server.webapp.WebApp.start(WebApp.java:1871)

note1:
I get a bunch of these, one for every mbean

note2:
I did notice the javax.management.InstanceNotFoundException that suggests it
means the mentioned Mbean doesn't exist. However if i enable jmx for any one
of my applications everything works. However if i enable it for more then
one then one webapp it starts giving these errors for every webapp after the
first that gets loaded. That and my experiences with fixing this same error
under Blaze DS suggests to me a very high likelyhood of some sort of name
collision going on.

Constantijn


On Tue, Mar 23, 2010 at 11:23 PM, Chris Hostetter
wrote:

>
> : I'm having a problem trying to get multiple solr applications to run in
> the
> : same servlet container because they all try to claim "solr" as a
>
> Hmmm... i think you're in new territory here.   I don't know that anyone
> has ever mentioned doing this before.
>
> Honestly: I thought the hierarchical nature of JMX would mean that
> the Servlet Container would start up a JMX server, and present a seperate
> "branch" to each webapp in isolation -- based on what you're saying it
> sounds like different webapps can't actually break eachother by mucking
> with JMX Beans/values.
>
> : If a configuration option like  exists that'd fix my
> : problem but i can't seem to find it in the documentation.
>
> It doesn't, but it would probably be pretty trivial to add if you want to
> take a stab at a patch for it.
>
>
> -Hoss
>
>

Re: Configuring multiple SOLR apps to play nice with MBeans / JMX

[Constantijn Visinescu]

Hi,

Multicore lets me have me have multiple cores in a single instance.

However since i have 3 different webapps with embedded solr that means i
have 3 different instances of solr.
(and they're all trying to park their JMX MBeans under the same name, namely
solr)

Constantijn


On Tue, Mar 23, 2010 at 11:44 AM, Charl Mert wrote:

> Hi Constantijn,
>
> I'm not too sure about the JMX monitoring side of things but having looked
> at the Solr's MultiCore <http://wiki.apache.org/solr/CoreAdmin>
> feature it seems really simple to create multiple solr cores that could all
> be configured to point
> to one MBean server.
>
> When creating a core you can specify name like solr1, solr2:
>
> http://localhost:8983/solr/admin/cores?action=CREATE&name=solr_01&instanceDir=/etc/solr/multicore/core2&config=solrconfig.xml&schema=schema.xml&dataDir=data
>
> This is made possible due to the fact that each core can have it's own
> solrconfig.xml
> See example/multicore/ in your solr distribution.
>
> Hope this helps.
>
> Regards
> Charl Mert
>
>
>
> On Tue, Mar 23, 2010 at 12:10 PM, Constantijn Visinescu
> wrote:
>
> > Hi,
> >
> > I'm having a problem trying to get multiple solr applications to run in
> the
> > same servlet container because they all try to claim "solr" as a
> > name/category to put their mbeans under and that causes
> exceptions/crashes
> > for all the applications after the first.
> >
> > I've read http://wiki.apache.org/solr/SolrJmx and it shows configuration
> > options to define a JMX server agentID or to provide your own JMX url but
> i
> > don't want either. (i think)
> >
> > I just want my webapps to show as "solr1", "solr2" and "solr3" when
> > monitoring them rather then all of them trying to race for "solr" and
> > having
> > all of them after the first crash.
> >
> > Right now I've disabled JMX and that works to get my apps started at
> least,
> > but it's not what i want either.
> >
> > Anyone know how to configure solr to do this?
> > If a configuration option like  exists that'd fix my
> > problem but i can't seem to find it in the documentation.
> >
> > Thanks in advance,
> > Constantijn Visinescu
> >
>

Configuring multiple SOLR apps to play nice with MBeans / JMX

[Constantijn Visinescu]

Hi,

I'm having a problem trying to get multiple solr applications to run in the
same servlet container because they all try to claim "solr" as a
name/category to put their mbeans under and that causes exceptions/crashes
for all the applications after the first.

I've read http://wiki.apache.org/solr/SolrJmx and it shows configuration
options to define a JMX server agentID or to provide your own JMX url but i
don't want either. (i think)

I just want my webapps to show as "solr1", "solr2" and "solr3" when
monitoring them rather then all of them trying to race for "solr" and having
all of them after the first crash.

Right now I've disabled JMX and that works to get my apps started at least,
but it's not what i want either.

Anyone know how to configure solr to do this?
If a configuration option like  exists that'd fix my
problem but i can't seem to find it in the documentation.

Thanks in advance,
Constantijn Visinescu

Re: Architectural help

[Constantijn Visinescu]

Assuming you create the view in such a way that it returns 1 row for each
solrdocument you want indexed: yes

On Wed, Mar 10, 2010 at 7:54 PM, blargy  wrote:

>
> So I can just create a view  (or temporary table) and then just have a
> simple
> "select * from (view or table)" in my DIH config?
>
>
> Constantijn Visinescu wrote:
> >
> > Try making a database view that contains everything you want to index,
> and
> > then just use the DIH.
> >
> > Worked when i tested it ;)
> >
> > On Wed, Mar 10, 2010 at 1:56 AM, blargy  wrote:
> >
> >>
> >> I was wondering if someone could be so kind to give me some
> architectural
> >> guidance.
> >>
> >> A little about our setup. We are RoR shop that is currently using Ferret
> >> (no
> >> laughs please) as our search technology. Our indexing process at the
> >> moment
> >> is quite poor as well as our search results. After some deliberation we
> >> have
> >> decided to switch to Solr to satisfy our search requirements.
> >>
> >> We have about 5M records ranging in size all coming from a DB source
> >> (only
> >> 2
> >> tables). What will be the most efficient way of indexing all of these
> >> documents? I am looking at DIH but before I go down that road I wanted
> to
> >> get some guidance. Are there any pitfalls I should be aware of before I
> >> start? Anything I can do now that will help me down the road?
> >>
> >> I have also been exploring the Sunspot rails plugin
> >> (http://outoftime.github.com/sunspot/) which so far seems amazing.
> There
> >> is
> >> an easy way to reindex all of your models like Model.reindex but I doubt
> >> this is the most efficient. Has anyone had any experience using Sunspot
> >> with
> >> their rails environment and if so should I bother with the DIH?
> >>
> >> Please let me know of any suggestions/opinions you may have. Thanks.
> >>
> >>
> >> --
> >> View this message in context:
> >> http://old.nabble.com/Architectural-help-tp27844268p27844268.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Architectural-help-tp27844268p27854256.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Architectural help

[Constantijn Visinescu]

Try making a database view that contains everything you want to index, and
then just use the DIH.

Worked when i tested it ;)

On Wed, Mar 10, 2010 at 1:56 AM, blargy  wrote:

>
> I was wondering if someone could be so kind to give me some architectural
> guidance.
>
> A little about our setup. We are RoR shop that is currently using Ferret
> (no
> laughs please) as our search technology. Our indexing process at the moment
> is quite poor as well as our search results. After some deliberation we
> have
> decided to switch to Solr to satisfy our search requirements.
>
> We have about 5M records ranging in size all coming from a DB source (only
> 2
> tables). What will be the most efficient way of indexing all of these
> documents? I am looking at DIH but before I go down that road I wanted to
> get some guidance. Are there any pitfalls I should be aware of before I
> start? Anything I can do now that will help me down the road?
>
> I have also been exploring the Sunspot rails plugin
> (http://outoftime.github.com/sunspot/) which so far seems amazing. There
> is
> an easy way to reindex all of your models like Model.reindex but I doubt
> this is the most efficient. Has anyone had any experience using Sunspot
> with
> their rails environment and if so should I bother with the DIH?
>
> Please let me know of any suggestions/opinions you may have. Thanks.
>
>
> --
> View this message in context:
> http://old.nabble.com/Architectural-help-tp27844268p27844268.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: DIH out of memory exception

[Constantijn Visinescu]

Does this help?

http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F

On Wed, Oct 28, 2009 at 12:38 AM, William Pierce wrote:

> Hi, Gilbert:
>
> Thanks for your tip!  I just tried it.  Unfortunately, it does not work for
> me.  I still get the OOM exception.
>
> How large was your dataset?  And what were your machine specs?
>
> Cheers,
>
> - Bill
>
> --
> From: "Gilbert Boyreau" 
> Sent: Tuesday, October 27, 2009 11:54 AM
> To: 
> Subject: Re: DIH out of memory exception
>
>
>  Hi,
>>
>> I got the same problem using DIH with a large dataset in MySql database.
>>
>> Following :
>> http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-implementation-notes.html
>> ,
>> and looking at the java code, it appears that DIH use PreparedStatement in
>> the JdbcDataSource.
>>
>> I set the batchsize parameter to -1 and it solved my problem.
>>
>> Regards.
>> Gilbert.
>>
>> William Pierce a écrit :
>>
>>> Folks:
>>>
>>> My db contains approx 6M records -- on average each is approx 1K bytes.
>>> When I use the DIH,  I reliably get an OOM exception.   The machine has 4 GB
>>> ram,  my tomcat is set to use max heap of 2G.
>>> The option of increasing memory is not tenable coz as the number of
>>> documents grows I will be back in this situation.
>>> Is there a way to batch the documents?  I tried setting the batchsize
>>> parameter to 500 on the  tag where I specify the jdbc
>>> parameters.   This had no effect.
>>>
>>> Best,
>>>
>>> - Bill
>>>
>>>
>>
>>

Re: Problem getting Solr home from JNDI in Tomcat

[Constantijn Visinescu]

This might be a bit of a hack but i got this in the web.xml of my applicatin
and it works great.



   solr/home
   /Solr/WebRoot/WEB-INF/solr
   java.lang.String


On Tue, Sep 29, 2009 at 2:32 PM, Andrew Clegg wrote:

>
> Hi all, I'm having problems getting Solr to start on Tomcat 6.
>
> Tomcat is installed in /opt/apache-tomcat , solr is in
> /opt/apache-tomcat/webapps/solr , and my Solr home directory is /opt/solr .
> My config file is in /opt/solr/conf/solrconfig.xml .
>
> I have a Solr-specific context file in
> /opt/apache-tomcat/conf/Catalina/localhost/solr.xml which looks like this:
>
> 
> value="/opt/solr" override="true" />
> allow="128\.40\.46\..*,127\.0\.0\.1" />
> 
>
> But when I start Solr and browse to it, it tells me:
>
> java.lang.RuntimeException: Can't find resource 'solrconfig.xml' in
> classpath or 'solr/conf/', cwd=/ at
>
> org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:194)
> at
>
> org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:162)
> at org.apache.solr.core.Config.(Config.java:100) at
> org.apache.solr.core.SolrConfig.(SolrConfig.java:113) at
> org.apache.solr.core.SolrConfig.(SolrConfig.java:70) at
>
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:117)
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> at
>
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
> at
>
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
> at
>
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108)
> at
>
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4356)
> at
> org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1244)
> at
>
> org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:604)
> at
>
> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:129)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
> at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> at
>
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
> at
>
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
> at
>
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
> at
>
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
> at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> at
>
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> at java.lang.Thread.run(Thread.java:619)
>
> Weirdly, the exact same context file works fine on a different machine.
> I've
> tried giving Context a docBase element (both absolute, and relative paths)
> but it makes no difference -- Solr still isn't seeing the right home
> directory. I also tried setting debug="1" but didn't see any more useful
> info anywhere.
>
> Any ideas? This is a total show-stopper for me as this is our production
> server. (Otherwise I'd think about taking it down and hardwiring the Solr
> home path into the server's context...)
>
> Yours hopefully,
>
> Andrew.
>
> --
> View this message in context:
> http://www.nabble.com/Problem-getting-Solr-home-from-JNDI-in-Tomcat-tp25662200p25662200.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Solr http post performance seems slow - help?

[Constantijn Visinescu]

This may or may not help but here goes :)

When i was running performance tests i look a look at the simple post tool
that comes with the solr examples.

First i changed my schema.xml to fit my needs and then i deleted the old
index so solr created a blank one when i started up.
Then i had a had a process chew on my data and spit out xml files that are
formatted similarly to the xml files that the SimplePostTool example uses.
Next i used the simple Post tool to post the xml files to solr (60k-80k
records per xml file). Each file only took a couple minutes to index this
way.
Comit and optimize after that (took less then 10 minutes) and after about
2.5 hrs i had indexed just under 8 milion records.

This was on a 4 year old single core laptop using resin 3 as my servlet
container.

Hope this helps.


On Fri, Sep 25, 2009 at 3:51 AM, Lance Norskog  wrote:

> In "top", press the '1' key. This will give a list of the CPUs and how
> much load is on each. The display is otherwise a little weird for
> multi-cpu machines. But don't be surprised when Solr is I/O bound. The
> biggest fanciest RAID is often a better investment than CPUs. On one
> project we bought low-end rack servers come with 6-8 disk bays,
> filling them with 10k/15k RPM disks.
>
> On Wed, Sep 23, 2009 at 2:47 PM, Dan A. Dickey 
> wrote:
> > On Friday 11 September 2009 11:06:20 am Dan A. Dickey wrote:
> > ...
> >> Our JBoss expert and I will be looking into why this might be occurring.
> >> Does anyone know of any JBoss related slowness with Solr?
> >> And does anyone have any other sort of suggestions to speed indexing
> >> performance?   Thanks for your help all!  I'll keep you up to date with
> >> further progress.
> >
> > Ok, further progress... just to keep any interested parties up to date
> > and for the record...
> >
> > I'm finding that using the "example" jetty setup (will be switching very
> > very soon to a "real" jetty installation) is about the fastest.  Using
> > several processes to send posts to Solr helps a lot, and we're seeing
> > about 80 posts a second this way.
> >
> > We also stripped down JBoss to the bare bones and the Solr in it
> > is running nearly as fast - about 50 posts a second.  It was our previous
> > JBoss configuration that was making it appear "slow" for some reason.
> >
> > We will be running more tests and spreading out the "pre-index" workload
> > across more machines and more processes. In our case we were seeing
> > the bottleneck being one machine running 18 processes.
> > The 2 quad core xeon system is experiencing about a 25% cpu load.
> > And I'm not certain, but I think this may be actually 25% of one of the 8
> cores.
> > So, there's *lots* of room for Solr to be doing more work there.
> >-Dan
> >
> > --
> > Dan A. Dickey | Senior Software Engineer
> >
> > Savvis
> > 10900 Hampshire Ave. S., Bloomington, MN  55438
> > Office: 952.852.4803 | Fax: 952.852.4951
> > E-mail: dan.dic...@savvis.net
> >
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>

Re: OutOfMemory error on solr 1.3

[Constantijn Visinescu]

1,5 GB already seems like quite a bit but adding more just might solve it
... try something like 3GB (if your machine supports it) and see if that
helps?

If 3 GB still doesn't cut it then the problem is most likely somewhere else
and i'd suggest looking at the application with a memory profiler to see if
something is wrong.

I see you're running jrockit .. does that mean you also have mission control
? I'm making assumptions here but i believe one of mission control's
features is that it lets you monitor the application, maybe that can tell
you something usefull .. otherwise grab a real memory profiler to find out
where all the memory is going.

On Thu, Sep 10, 2009 at 7:44 PM, Francis Yakin  wrote:

> SO, do you think increasing the JVM will help? We also have
> 500 in solrconfig.xml
> Originally was set to
> 200
>
> Currently we give solr 1.5GB for Xms and Xmx, we use jrockit version
> 1.5.0_15
>
> 4 S root 12543 12495 16  76   0 - 848974 184466 Jul20 ?
> 8-11:12:03 /opt/bea/jrmc-3.0.3-1.5.0/bin/java -Xms1536m -Xmx1536m -Xns:128m
> -Xgc:gencon -Djavelin.jsp.el.elcache=4096
> -Dsolr.solr.home=/opt/apache-solr-1.3.0/example/solr
>
> Francis
>
> -Original Message-
> From: Constantijn Visinescu [mailto:baeli...@gmail.com]
> Sent: Wednesday, September 09, 2009 11:35 PM
> To: solr-user@lucene.apache.org
> Subject: Re: OutOfMemory error on solr 1.3
>
> Just wondering, how much memory are you giving your JVM ?
>
> On Thu, Sep 10, 2009 at 7:46 AM, Francis Yakin  wrote:
>
> >
> > I am having OutOfMemory error on our slaves server, I would like to know
> if
> > someone has the same issue and have the solution for this.
> >
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@96cd2ffc
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> > SEVERE: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object
> size:
> > 441216, Num elements: 55150
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@519116e0
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@74dc52fa
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@d0dd3e28
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@b6dfa5bc
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@482b13ef
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@2309438c
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@277bd48c
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> > Exception in thread "[ACTIVE] ExecuteThread: '7' for queue:
> > 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> > Exception in thread "[ACTIVE] ExecuteThread: '8' for queue:
> > 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> > Exception in thread "[ACTIVE] ExecuteThread: '10' for queue:
> > 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> > Exception in thread "[ACTIVE] ExecuteThread: '11' for queue:
> > 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> > SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.queryresult...@41405463
> :java.lang.OutOfMemoryError:
> > allocLargeObjectOrArray - Object size: 751552, Num elements: 187884
> >  java.lang.OutOfMemoryError: allocLargeObjectOrArray - Obj

Re: Solr fitting in travel site context?

[Constantijn Visinescu]

I'd run look into faceting and run a test.

Create a schema, index the data and then run a query for *:* facteted by
hotel to get a list of all the hotels you want followed by a query that
returns all documents matching that hotel for your 2nd usecase.

You're probably still going to want a SQL database to catch the reservations
made tho.

in my experience implementing Solr is more work then implementing a normal
SQL database, and loosing the relational part of a relational database is
something you have to wrap your head around to see how it affects your
application.

That said solr on my 4 year old single core laptop outperforms our new dual
xeon database server running IBM DB2 when it comes to running a query on a
10 million record dataset and retuning the total amount of documents that
match.

Once you get it up and running properly and you need querys that are like
"give me the total number of documents that match these criteria, optionally
facted by this and that" it's amazingly fast.

Note that this advantage only becomes apparent when dealing with large data
sets. anything under a coulpe 100k records (guideline, depends heavily on
the type of record) and a normal SQL server should also be able to give you
the results you need near instantly.

Hope this helps ;)


On Wed, Sep 9, 2009 at 5:33 PM, Carsten Kraus wrote:

> Hi all,
>
> I'm about to develop a travel website and am wondering if Solr might fit to
> be used as the search solution.
> Being quite the opposite of a db guru and new to Solr, it's hard for me to
> judge if for my use-case a relational db should be used in favor of Solr(or
> similar indexing server). Maybe some of you guys would share their opinion
> on this?
>
> The products being searched for would be travel packages. That is: hotel
> room + flight combined into one product.
> I receive the products via a csv file, where each line defines a travel
> package with concrete departure/return, accommodation and price data.
>
> For example one csv row might represent:
> Hotel Foo in Paris, flight departing 10/10/09 from London, ending 10/20/09,
> mealplan Bar, pricing $300
> ..while another one might look like:
> Hotel Foo in Paris, flight departing 10/10/09 from Amsterdam, ending
> 10/30/09, mealplan Eggs :), pricing $400
>
> Now searches should show results in 2 steps: first step showing results
> grouped by hotel(so no hotel appears twice) and second one all
> date-airport-mealplan combinations for the hotel selected by the user in
> step 1.
>
> From some first little tests, it seems to me as if I at least would need
> the
> collapse patch(SOLR-236) to be used in step 1 above?!
>
> What do you think? Does Solr fit into this scenario? Thoughts?
>
> Sorry for the lengthy post & thanks a lot for any pointer!
> Carsten
>

Re: OutOfMemory error on solr 1.3

[Constantijn Visinescu]

Just wondering, how much memory are you giving your JVM ?

On Thu, Sep 10, 2009 at 7:46 AM, Francis Yakin  wrote:

>
> I am having OutOfMemory error on our slaves server, I would like to know if
> someone has the same issue and have the solution for this.
>
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@96cd2ffc:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> SEVERE: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size:
> 441216, Num elements: 55150
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@519116e0:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@74dc52fa:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@d0dd3e28:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@b6dfa5bc:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@482b13ef:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@2309438c:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@277bd48c:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14128832, Num elements: 3532204
> Exception in thread "[ACTIVE] ExecuteThread: '7' for queue:
> 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> Exception in thread "[ACTIVE] ExecuteThread: '8' for queue:
> 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> Exception in thread "[ACTIVE] ExecuteThread: '10' for queue:
> 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> Exception in thread "[ACTIVE] ExecuteThread: '11' for queue:
> 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 8208, Num elements: 8192
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@41405463:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 751552, Num elements: 187884
>  java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 8208,
> Num elements: 8192
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 8208,
> Num elements: 8192
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 5096,
> Num elements: 2539
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 5400,
> Num elements: 2690
>
>  deployment service message for request id "-1" from server "AdminServer".
> Exception is: "java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object
> size: 4368, Num elements: 2174
> SEVERE: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size:
> 14140768, Num elements: 3535188
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@8dbcc7ab:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5395576, Num elements: 1348890
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 5320,
> Num elements: 2649
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@4d0c6fc5:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 751560, Num elements: 187885
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 16400,
> Num elements: 8192
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@fb6bac19:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14140904, Num elements: 3535222
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@536d7b1b:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 14140904, Num elements: 3535222
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@6a1ef00a:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 751864, Num elements: 187961
> SEVERE: Error during auto-warming of
> key:org.apache.solr.search.queryresult...@298f2d9c:java.lang.OutOfMemoryError:
> allocLargeObjectOrArray - Object size: 5398568, Num elements: 1349637
> SEVERE: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object si

Re: slow response

[Constantijn Visinescu]

Just wondering, is there an easy way to load the whole index into ram?

On Wed, Sep 9, 2009 at 4:22 PM, Alex Baranov wrote:

> There is a good article on how to scale the Lucene/Solr solution:
>
>
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr
>
> Also, if you have heavy load on the server (large amount of concurrent
> requests) then I'd suggest to consider loading the index into RAM. It
> worked
> well for me on the project with 140+ million documents and 30 concurrent
> user requests per second. If your index can be placed in RAM you can reduce
> the architecture complexity.
>
> Alex Baranov
>
> On Wed, Sep 9, 2009 at 5:10 PM, Elaine Li 
> wrote:
>
> > Hi,
> >
> > I have 20 million docs on solr. If my query would return more than
> > 10,000 results, the response time will be very very long. How to
> > resolve such problem? Can I slice my docs into pieces and let the
> > query operate within one piece at a time so the response time and
> > response data will be more managable? Thanks.
> >
> > Elaine
> >
>

Re: Solr Query help - sorting

[Constantijn Visinescu]

make a new multivalued field in your schema.xml, copy both width and length
into that field, and then sort on that field ?

On Tue, Aug 25, 2009 at 5:40 AM, erikea...@yahoo.com wrote:

> Clever... but if more than one row adds up to the same value I may get the
> wrong order (like 50, 50 and 10, 90)
>
> I need a max function but the one provided only compares against a
> constant.
>
> Sent from my iPhone
>
> On Aug 24, 2009, at 6:15 PM, Koji Sekiguchi  wrote:
>
> How about using sum() FunctionQuery, via:
>
> http://localhost:8983/solr/select/?q=_val_%3A%22sum%28width%2Clength%29%22
>
> Thanks,
>
> Koji
>
>
> Erik Earle wrote:
> Suppose I have a database of rectangle attributes that I have indexed in
> Solr and I want to get the top 10 widest or longest rectangles from one
> query.
>
>
> name  |  width  |  length
> A |  100|  10
> B |  10 |  90
> C |  80 |  10
> ...
>
>
> My indexed fields are:  name, width, length
>
> Is there a way to query so that my data returns the order:  A, B, C
>
>
>
>
>
>
>
>
>
>

Re: Optimizing a query to sort results alphabetically for a determinated field

[Constantijn Visinescu]

not 100% sure  but the example schema has:


So i'd say give it a go and see what happens ;)

On Mon, Aug 24, 2009 at 12:24 PM, Marc Sturlese wrote:

>
> Yes but I thought it was just for sortable fields:
> sint,sfloat,sdouble,slong.
> Can I apply "sortMissingLast"to  text fields analyzed with KeywordAnalyzer?
>
> Constantijn Visinescu wrote:
> >
> > There's a "sortMissingLast" true/false property that you can set on your
> > fielType definitions in the schema.
> >
> > On Mon, Aug 24, 2009 at 11:58 AM, Marc Sturlese
> > wrote:
> >
> >>
> >> Hey there, I need to sort my query results alphabetically for a
> >> determinated
> >> field called "town". This field is analyzed with a KeywordAnalyzer and
> >> isn't
> >> multiValued. Add that some docs doesn't doesn'h have this field.
> >> Doing just:
> >>
> >>
> >>
> http://localhost/solr//select/?q=whatever&version=2.2&start=0&rows=10&indent=on&sort=town
> >> asc
> >>
> >> Will give me back the results sorted alphabetically but will put the
> docs
> >> that doesn't have this field (town) at the begining.
> >> I want them at the end or I want them not to apear. This query solves
> the
> >> problem:
> >>
> >>
> >>
> http://localhost/solr//select/?q=whatever&version=2.2&start=0&rows=10&indent=on&sort=town
> >> asc&fq=town:[a<
> http://localhost/solr//select/?q=whatever&version=2.2&start=0&rows=10&indent=on&sort=town%0Aasc&fq=town:%5Ba
> >TO
> >> z]
> >>
> >> But applying this filter: fq=town:[a TO z] is definitely not good in
> >> terms
> >> of memory, speed and clauses...
> >> Is there any way to do something similar but with a more optimized
> query?
> >> Thanks in advance!
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Optimizing-a-query-to-sort-results-alphabetically-for-a-determinated-field-tp25113379p25113379.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Optimizing-a-query-to-sort-results-alphabetically-for-a-determinated-field-tp25113379p25113637.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Optimizing a query to sort results alphabetically for a determinated field

[Constantijn Visinescu]

There's a "sortMissingLast" true/false property that you can set on your
fielType definitions in the schema.

On Mon, Aug 24, 2009 at 11:58 AM, Marc Sturlese wrote:

>
> Hey there, I need to sort my query results alphabetically for a
> determinated
> field called "town". This field is analyzed with a KeywordAnalyzer and
> isn't
> multiValued. Add that some docs doesn't doesn'h have this field.
> Doing just:
>
>
> http://localhost/solr//select/?q=whatever&version=2.2&start=0&rows=10&indent=on&sort=town
> asc
>
> Will give me back the results sorted alphabetically but will put the docs
> that doesn't have this field (town) at the begining.
> I want them at the end or I want them not to apear. This query solves the
> problem:
>
>
> http://localhost/solr//select/?q=whatever&version=2.2&start=0&rows=10&indent=on&sort=town
> asc&fq=town:[aTO
>  z]
>
> But applying this filter: fq=town:[a TO z] is definitely not good in terms
> of memory, speed and clauses...
> Is there any way to do something similar but with a more optimized query?
> Thanks in advance!
>
> --
> View this message in context:
> http://www.nabble.com/Optimizing-a-query-to-sort-results-alphabetically-for-a-determinated-field-tp25113379p25113379.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Remove data from index

[Constantijn Visinescu]

You could write a solr query that queries *:* and only returns the id field
and then throw out all the IDs from "select id from databaseTable"
and then run a delete query for all the IDs that are left after wards.

However you'd have to write a seperate program/script to do this i think as
the DIH won't be able to do this my itself.



On Thu, Aug 20, 2009 at 5:09 PM, clico  wrote:

>
> I hope it could be a solution.
>
> But I think I understood that u can use deletePkQuery like this
>
> "select document_id from table_document where statusDeleted= 'Y'"
>
> In my case I have no status like "statusDeleted".
>
> The request I would like to write is
>
> "Delete from my solr Index the id that are no longer present in my
> table_document"
>
> With Lucene I had a way to do that :
> open IndexReader,
> for each lucene document : check in table_document and remove in lucene
> index if document is no longer present in the table
>
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Remove-data-from-index-tp25063736p25063965.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Group by field in Solr

[Constantijn Visinescu]

You'll want to use faceting, try to use a query like this

http://localhost:8080/Solr/select/?q=artist%3Awar&version=2.2&start=0&rows=0&facet=true&facet.limit=-1&facet.field=artist

replace localhost:8080 with your own :)

On Thu, Aug 20, 2009 at 2:40 PM, Daniel Löfquist <
daniel.lofqu...@it.cdon.com> wrote:

> Hello,
>
> I'm trying to accomplish something akin to "GROUP BY" in SQL but in Solr.
> I have an index full of songs (one song per document in Solr) by various
> artists and I
> would like to construct a search that gives me all of the artists back, one
> row per
> artist. The current search returns one row per artist and song.
>
> So I get this right now if I search after "war" in the artist-field:
>
> 30 Years War - Ideal Means
> 30 Years War - Dirty Castle
> 30 Years War - Misinformed
> All Out War - Soaked In Torment
> All Out War - Claim Your Innocence
> Audio War - Negativity
> Audio War - One Drug
> Audio War - Super Freak
>
> But this is what I'd really like:
>
> 30 Years War - whatever song
> All Out War - whatever song
> Audio War - whatever song
>
> I tried using facets but couldn't get it to work properly. Anybody have a
> clue how to do
> something like this?
>
> //Daniel
>

Re: dynamic changes to schema

[Constantijn Visinescu]

There's that or you can just change the user entered "my_author" field into
"my_author_customattribute" in code after the user has entered it and add a
*_customattribute to your schema.

you'd have to add the postfix in code also at querytime and off you go.

Constantijn

On Wed, Aug 19, 2009 at 11:52 PM, Erik Hatcher wrote:

> However, you can have a dynamic "*" field mapping that catches all field
> names that aren't already defined - though all of the fields will be the
> same field type.
>
>Erik
>
>
>
>
> On Aug 19, 2009, at 5:48 PM, Marco Westermann wrote:
>
>  Hi, thanks for your answers, I think I have to go more in deatail.
>>
>> we are talking about a shop-application which have products I want to
>> search for. This products normally have the standard attributes like sku, a
>> name, a price and so on. But the user can add attributes to the product. So
>> for example if he sells books, he could add the author as attribute. Lets
>> say he name this field my_author (but he is free to name it as he wants) and
>> he tells this field over  the configuration, that it is searchable. So I
>> need a field in solr for the author. Cause I cant restrict the user to
>> prefix every field with something like my_ dynamic fields doesn't work, do
>> they?
>>
>> best,
>> Marco
>>
>> Constantijn Visinescu schrieb:
>>
>>> huh? I think I lost you :)
>>> You want to use a multivalued field to list what dynamic fields you have
>>> in
>>> your document?
>>>
>>> Also if you program your application correctly you should be able to
>>> restrict your users from doing anything you please (or don't please in
>>> this
>>> case).
>>>
>>>
>>> On Tue, Aug 18, 2009 at 11:38 PM, Marco Westermann 
>>> wrote:
>>>
>>>
>>>  hi,
>>>>
>>>> thanks for the advise but the problem with dynamic fields is, that i
>>>> cannot
>>>> restrict how the user calls the field in the application. So there isn't
>>>> a
>>>> pattern I can use. But I thought about using mulitvalued fields for the
>>>> dynamically added fields. Good Idea?
>>>>
>>>> thanks,
>>>> Marco
>>>>
>>>> Constantijn Visinescu schrieb:
>>>>
>>>>
>>>>  use a dynamic field ?
>>>>>
>>>>> On Tue, Aug 18, 2009 at 5:09 PM, Marco Westermann 
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  Hi there,
>>>>>>
>>>>>> is there a possibility to change the solr-schema over php dynamically.
>>>>>> The
>>>>>> web-application I want to index at the moment has the feature to add
>>>>>> fields
>>>>>> to entitys and you can tell this fields that they are searchable. To
>>>>>> realize
>>>>>> this with solr the schema has to change when a searchable field is
>>>>>> added
>>>>>> or
>>>>>> removed.
>>>>>>
>>>>>> Any suggestions,
>>>>>>
>>>>>> Thanks a lot,
>>>>>>
>>>>>> Marco Westermann
>>>>>>
>>>>>>

Re: dynamic changes to schema

[Constantijn Visinescu]

huh? I think I lost you :)
You want to use a multivalued field to list what dynamic fields you have in
your document?

Also if you program your application correctly you should be able to
restrict your users from doing anything you please (or don't please in this
case).


On Tue, Aug 18, 2009 at 11:38 PM, Marco Westermann  wrote:

> hi,
>
> thanks for the advise but the problem with dynamic fields is, that i cannot
> restrict how the user calls the field in the application. So there isn't a
> pattern I can use. But I thought about using mulitvalued fields for the
> dynamically added fields. Good Idea?
>
> thanks,
> Marco
>
> Constantijn Visinescu schrieb:
>
>> use a dynamic field ?
>>
>> On Tue, Aug 18, 2009 at 5:09 PM, Marco Westermann 
>> wrote:
>>
>>
>>
>>> Hi there,
>>>
>>> is there a possibility to change the solr-schema over php dynamically.
>>> The
>>> web-application I want to index at the moment has the feature to add
>>> fields
>>> to entitys and you can tell this fields that they are searchable. To
>>> realize
>>> this with solr the schema has to change when a searchable field is added
>>> or
>>> removed.
>>>
>>> Any suggestions,
>>>
>>> Thanks a lot,
>>>
>>> Marco Westermann
>>>
>>> --
>>> ++ Business-Software aus einer Hand ++
>>> ++ Internet, Warenwirtschaft, Linux, Virtualisierung ++
>>> http://www.intersales.de
>>> http://www.eisxen.org
>>> http://www.tarantella-partner.de
>>> http://www.medisales.de
>>> http://www.eisfair.net
>>>
>>> interSales AG Internet Commerce
>>> Subbelrather Str. 247
>>> 50825 Köln
>>>
>>> Tel  02 21 - 27 90 50
>>> Fax  02 21 - 27 90 517
>>> Mail i...@intersales.de
>>> Mail m...@intersales.de
>>> Web  www.intersales.de
>>>
>>> Handelsregister Köln HR B 30904
>>> Ust.-Id.: DE199672015
>>> Finanzamt Köln-Nord. UstID: nicht vergeben
>>> Aufsichtsratsvorsitzender: Michael Morgenstern
>>> Vorstand: Andrej Radonic, Peter Zander
>>>
>>>
>>>
>>
>>
>>
>> __ Hinweis von ESET NOD32 Antivirus, Signaturdatenbank-Version
>> 4346 (20090818) __
>>
>> E-Mail wurde geprüft mit ESET NOD32 Antivirus.
>>
>> http://www.eset.com
>>
>>
>>
>>
>
>
> --
> ++ Business-Software aus einer Hand ++
> ++ Internet, Warenwirtschaft, Linux, Virtualisierung ++
> http://www.intersales.de
> http://www.eisxen.org
> http://www.tarantella-partner.de
> http://www.medisales.de
> http://www.eisfair.net
>
> interSales AG Internet Commerce
> Subbelrather Str. 247
> 50825 Köln
>
> Tel  02 21 - 27 90 50
> Fax  02 21 - 27 90 517
> Mail i...@intersales.de
> Mail m...@intersales.de
> Web  www.intersales.de
>
> Handelsregister Köln HR B 30904
> Ust.-Id.: DE199672015
> Finanzamt Köln-Nord. UstID: nicht vergeben
> Aufsichtsratsvorsitzender: Michael Morgenstern
> Vorstand: Andrej Radonic, Peter Zander
>

Re: dynamic changes to schema

[Constantijn Visinescu]

use a dynamic field ?

On Tue, Aug 18, 2009 at 5:09 PM, Marco Westermann  wrote:

> Hi there,
>
> is there a possibility to change the solr-schema over php dynamically. The
> web-application I want to index at the moment has the feature to add fields
> to entitys and you can tell this fields that they are searchable. To realize
> this with solr the schema has to change when a searchable field is added or
> removed.
>
> Any suggestions,
>
> Thanks a lot,
>
> Marco Westermann
>
> --
> ++ Business-Software aus einer Hand ++
> ++ Internet, Warenwirtschaft, Linux, Virtualisierung ++
> http://www.intersales.de
> http://www.eisxen.org
> http://www.tarantella-partner.de
> http://www.medisales.de
> http://www.eisfair.net
>
> interSales AG Internet Commerce
> Subbelrather Str. 247
> 50825 Köln
>
> Tel  02 21 - 27 90 50
> Fax  02 21 - 27 90 517
> Mail i...@intersales.de
> Mail m...@intersales.de
> Web  www.intersales.de
>
> Handelsregister Köln HR B 30904
> Ust.-Id.: DE199672015
> Finanzamt Köln-Nord. UstID: nicht vergeben
> Aufsichtsratsvorsitzender: Michael Morgenstern
> Vorstand: Andrej Radonic, Peter Zander
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

Am pretty sure solr.xml is if you want to define multiple solr
cores for your application.So it makes sense that solr checks for that
first, however if it doesn't find one it continues to start up with a single
core. 

I KNOW it runs just fine for me without a solr.xml.

The exception seems to be complaining about Xpath. Etiher this means that
you have a weird xml library on your classpath somewhere (unlikely but check
it, Xalan/Xerces seems to be the most common) or tomcat/solr doesn't have
the rights it needs to access the solr folder.

What rights did you give the solr folder and all the files in there ?

On Tue, Aug 18, 2009 at 3:27 PM, Aaron Aberg  wrote:

> Constantijn,
>
> First of all, I want you to know how much I appreciate you not giving
> up on me. Second of all, your instructions were really great. I think
> that I am getting closer to solving this issue. I am STILL get that
> error but after a full tomcat reboot it picked up my solr.home
> environment variable from my web.xml and its pointing to the new
> location. (Good idea)
>
> Here is the FULL log from start up of Tomcat. It might be excessive,
> but I want to give you all of the information that I can:
>
> Aug 17, 2009 11:16:08 PM org.apache.catalina.core.AprLifecycleListener
> lifecycleEvent
> INFO: The Apache Tomcat Native library which allows optimal
> performance in production environments was not found on the
> java.library.path:
>
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386/client:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
> Aug 17, 2009 11:16:09 PM org.apache.coyote.http11.Http11BaseProtocol init
> INFO: Initializing Coyote HTTP/1.1 on http-8080
> Aug 17, 2009 11:16:09 PM org.apache.coyote.http11.Http11BaseProtocol init
> INFO: Initializing Coyote HTTP/1.1 on http-9080
> Aug 17, 2009 11:16:09 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 3382 ms
> Aug 17, 2009 11:16:09 PM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Aug 17, 2009 11:16:09 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/5.5.23
> Aug 17, 2009 11:16:09 PM org.apache.catalina.core.StandardHost start
> INFO: XML validation disabled
> Aug 17, 2009 11:16:12 PM org.apache.catalina.core.ApplicationContext log
> INFO: ContextListener: contextInitialized()
> Aug 17, 2009 11:16:12 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: contextInitialized()
> Aug 17, 2009 11:16:12 PM org.apache.catalina.core.ApplicationContext log
> INFO: ContextListener: contextInitialized()
> Aug 17, 2009 11:16:12 PM org.apache.catalina.core.ApplicationContext log
> INFO: SessionListener: contextInitialized()
> Aug 17, 2009 11:16:12 PM org.apache.catalina.core.ApplicationContext log
> INFO: org.apache.webapp.balancer.BalancerFilter: init(): ruleChain:
> [org.apache.webapp.balancer.RuleChain:
> [org.apache.webapp.balancer.rules.URLStringMatchRule: Target string:
> News / Redirect URL: http://www.cnn.com],
> [org.apache.webapp.balancer.rules.RequestParameterRule: Target param
> name: paramName / Target param value: paramValue / Redirect URL:
> http://www.yahoo.com],
> [org.apache.webapp.balancer.rules.AcceptEverythingRule: Redirect URL:
> http://jakarta.apache.org]]
> Aug 17, 2009 11:16:13 PM org.apache.coyote.http11.Http11BaseProtocol start
> INFO: Starting Coyote HTTP/1.1 on http-8080
> Aug 17, 2009 11:16:13 PM org.apache.jk.common.ChannelSocket init
> INFO: JK: ajp13 listening on /0.0.0.0:8009
> Aug 17, 2009 11:16:13 PM org.apache.jk.server.JkMain start
> INFO: Jk running ID=0 time=0/57  config=null
> Aug 17, 2009 11:16:13 PM org.apache.catalina.core.StandardService start
> INFO: Starting service PSA
> Aug 17, 2009 11:16:13 PM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/5.5.23
> Aug 17, 2009 11:16:13 PM org.apache.catalina.core.StandardHost start
> INFO: XML validation disabled
> Aug 17, 2009 11:16:15 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> Aug 17, 2009 11:16:15 PM org.apache.solr.core.SolrResourceLoader
> locateInstanceDir
> INFO: Using JNDI solr.home: /usr/share/solr
> Aug 17, 2009 11:16:15 PM
> org.apache.solr.core.CoreContainer$Initializer initialize
> INFO: looking for solr.xml: /usr/share/solr/solr.xml
> Aug 17, 2009 11:16:15 PM org.apache.solr.core.SolrResourceLoader 
> INFO: Solr home set to '/usr/share/solr/'
> Aug 17, 2009 11:16:15 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Reusing parent classloader
> Aug 17, 2009 11:16:15 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
> java.lang.ExceptionInInitializerError
> at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:117)
> at
> org.apache.solr.servlet.So

Re: Release Date Solr 1.4

[Constantijn Visinescu]

Last i heard the eta was aprox a month, but they won't release it untill
it's ready.

Check JIRA here for the list of open issues that need fixing before 1.4
https://issues.apache.org/jira/secure/IssueNavigator.jspa?sorter/field=updated&sorter/order=DESC

Constantijn Visinescu

On Tue, Aug 18, 2009 at 2:57 PM, Daniel Knapp <
daniel.kn...@mni.fh-giessen.de> wrote:

> Hello Mailinglist,
>
>
> does anyone know the release date from Solr 1.4?
>
> Thanks for your reply.
>
>
> Regards,
> Daniel
>

Re: Maximum number of values in a multi-valued field.

[Constantijn Visinescu]

Hi,

This would also depend on the amount of documents you got in your solr
index.

10k ? 100k? 1m? 10m? 100m?

I'm by no means an expert on solr but i recently had a similar question and
to get my answer i grabbed a new blank solr index, modiefied my schema.xml,
reindexed all my data assuming the worst case.

Make up something like 100k (pick a number that seams realistic and multiply
by 10) different catalog names and assign 2000 to each document randomly.
Let your computer crunch bits for a few hours to rebuild an index (close to
10m documents in my case), copy the index to a server similar to your
production server and see what happens :)

For me there was no noticeable performance difference.

Constantijn Visinescu

On Tue, Aug 18, 2009 at 1:15 AM, Aravind Naidu  wrote:

>
> Hi,
> The possibility is that all items in this field could be unique. Let me
> clarify.
> The main Solr index is a for a list of products. Some products belong to
> catalogues.  So, the consideration is to add a multi-valued field to put
> the
> id of the catalogue in each product as a multi-valued field to be used as a
> filter.
>
> -- Aravind
>
>
> Jason Rutherglen-2 wrote:
> >
> > Your term dictionary will grow somewhat, which means the term
> > index could consume more memory. Because the term dictionary has
> > grown there could be less performance in looking up terms but
> > that is unlikely to affect your application. How many unique
> > terms will there be?
> >
> > On Mon, Aug 17, 2009 at 3:50 PM, Arv wrote:
> >>
> >> All,
> >> We are considering some new changes to our Solr schema to better support
> >> some new functionality for our application. To that extent, we want to
> >> add
> >> an additional field that is multi-valued, but will contain a large
> number
> >> of
> >> values per document. Potentially up to 2000 values on this field per
> >> document.
> >>
> >> Questions:
> >> - Is this wise?
> >> - Though we will not be faceting on this field, are there any
> >> implications
> >> for performance?
> >> - I understand that the XML in/out will be large, and we may need to
> stop
> >> this field being sent back on every query, as this field is essentially
> >> used
> >> as a filter only.
> >>
> >> The reason I am asking is that our instance of Solr currently works
> >> wonderfully and is very fast, and I am wary of doing anything that will
> >> affect this.  So, any pointer on design here will help.
> >>
> >> -- Aravind
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015685.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015945.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

Ok ... sounds like something is screwed up somewhere(tm). Hard part is
figuring out where :)


My suggestion:

Throw everything that's remotely related to the webapp you're trying to
build off that server and make sure you get all of it. (no stray bits of
solr confuration files leftover anywhere).

Next get apache-solr-1.3.0.zip from
http://apache.mirrors.webazilla.nl/lucene/solr/1.3.0/ (or whatever mirror
you prefer, see solr main page)

Make a directory for solr on your server somewhere that's completely
separate from anything tomcat or webserver related.(i.e. not a subdirectory
of usr/share/tomcat5/ ... maybe try making /usr/share/solr/ ? put it
somewhere it's guaranteed not to interfere with anything :))

Copy the contents of \apache-solr-1.3.0\example\solr\ to the directory you
just made using ssh

Throw away the data folder if it exists. (that should leave you with the bin
and conf folders + readme.txt, no lib folder or anything else)

in the conf folder comment out the
${solr.data.dir:./solr/data} line in solrconfig.xml near
the top.

make sure tomcat has all the rights it needs on the /usr/share/solr/ folder
(i'm assuming read/write, someone correct me if I'm wrong)

Grab apache-solr-1.3.0.war from apache-solr-1.3.0\dist\ and edit the web.xml
to include

   solr/home
   /usr/share/solr
   java.lang.String

near the end.

Upload the war with the new web.xml using plesk.

Surf to http://myserver:myport/apache-solr-1.3.0/index.jsp to see your solr
installation up and running.

After that solr should have created a new blank index in the
usr/shared/solr/data folder and solr should be up and running.

If this works you should be able to start adding all the bits and pieces of
your own app and go from there :) This might be a bit overkill but if you
can get this up and running you should be able to get it to do what you need
to do in the end. If this doesn't work double check you got the path in the
web.xml correct and that tomcat can access that folder with all rights. If
it still doesn't work after that I'm not sure what to do ;)


On Tue, Aug 18, 2009 at 5:34 AM, Funtick  wrote:

>
> It is NOT sample war, it is SOLR application: solr.war - it should be!!! I
> usually build from source and use dist/apache-solr-1.3.war instead, so I am
> not sure about solr.war
>
> solr.xml contains configuration for multicore; most probably something is
> wrong with it.
> Would be better if you try to install Tomcat on localbox and play with it
> before going to production...
>
>
>
> Aaron Aberg wrote:
> >
> > On Mon, Aug 17, 2009 at 10:58 AM, Fuad Efendi wrote:
> >> Looks like you are using SOLR multicore, with solr.xml... I never tried
> >> it...
> >> The rest looks fine, except suspicious solr.xml
> >
> > whats suspicious about it? is it in the wrong place? Is it not suppose
> > to be there?
> >
> > technically my war file is not in my webapps directory. I'm using
> > plesk and it installed my war file here:
> > tomcat5/psa-webapps/mywebk9.com/solr.war
> >
> > I have installed a sample war file and its in the same location. It works
> > fine.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Cannot-get-solr-1.3.0-to-run-properly-with-plesk-9.2.1-on-CentOS-tp24980824p25017895.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

s.txt
> /usr/share/tomcat5/solr/mywebk9.com/conf/admin-extra.html
> /usr/share/tomcat5/solr/mywebk9.com/conf/stopwords.txt
> /usr/share/tomcat5/solr/webapps
> /usr/share/tomcat5/solr/webapps/solr.war
>
> I'm still getting this error:
>
> Aug 16, 2009 6:14:44 PM org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
> locateInstanceDir
> INFO: Using JNDI solr.home: /usr/share/tomcat5/solr
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.CoreContainer$Initializer
> initialize
> INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader 
> INFO: Solr home set to '/usr/share/tomcat5/solr/'
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar' to
> Solr classloader
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar'
> to Solr classloader
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar'
> to Solr classloader
> Aug 16, 2009 6:14:44 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jsp-2.1/' to Solr
> classloader
> Aug 16, 2009 6:14:44 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
>
> I don't understand... it says solr home set to
> '/usr/share/tomcat5/solr/' but the error says that there is something
> wrong with the solr/home directory. Is my file structure incorrect and
> its been masked by this error?
>
> --Aaron
>
> On Mon, Aug 17, 2009 at 12:57 AM, Constantijn
> Visinescu wrote:
> > near the bottom of my web.xml (just above ) i got
> >
> >
> >   solr/home
> >   path/to/solr
> >   java.lang.String
> >
> >
> > While you're at it you might want to make sure the following line in your
> > solrconfig.xml is commented out
> >  
> >
> > next you should copy the sorl directory (the one with the conf, data and
> bin
> > subdiretories) to the path you specified in the web.xml
> >
> > Repackage the war and redeploy, that should do it :)
> >
> >
> > On Sat, Aug 15, 2009 at 11:59 AM, Aaron Aberg 
> wrote:
> >
> >> Ok, I'm still having a problem. I believe that I have to set the
> >> solr.solr.home variable somewhere. In Windows, that was easy because
> >> you could just open up the system tray app for tomcat and set it
> >> there. How can I do that in linux? After googling, it suggested that I
> >> set an environment variable in the web.xml file inside the war file. I
> >> even saw a commented out section for setting the home variable. I set
> >> it in there but it still is giving me this error:
> >>
> >> INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
> >> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader 
> >> INFO: Solr home set to 'solr/'
> >> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> >> createClassLoader
> >> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar' to
> >> Solr classloader
> >> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> >> createClassLoader
> >> INFO: Adding
> 'file:/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar'
> >> to Solr classloader
> >> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> >> createClassLoader
> >> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar'
> >> to Solr classloader
> >> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> >> createClassLoader
> >> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jsp-2.1/' to Solr
> >> classloader
> >> Aug 14, 2009 6:16:42 PM org.apache.solr.servlet.SolrDispatchFilter init
> >> SEVERE: Could not start SOLR. Check solr/home property
> >> java.lang.ExceptionInInitializerError
> >>
> >> Any ideas?
> >>
> >> --Aaron
> >>
> >> On Sat, Aug 15, 2009 at 12:40 AM, Shalin Shekhar
> >> Mangar wrote:
> >> > On Sat, Aug 15, 2009 at 6:52 AM, Aaron Aberg 
> >> wrote:
> >

Re: Cannot get solr 1.3.0 to run properly with plesk 9.2.1 on CentOS

[Constantijn Visinescu]

near the bottom of my web.xml (just above ) i got


   solr/home
   path/to/solr
   java.lang.String


While you're at it you might want to make sure the following line in your
solrconfig.xml is commented out
  

next you should copy the sorl directory (the one with the conf, data and bin
subdiretories) to the path you specified in the web.xml

Repackage the war and redeploy, that should do it :)


On Sat, Aug 15, 2009 at 11:59 AM, Aaron Aberg  wrote:

> Ok, I'm still having a problem. I believe that I have to set the
> solr.solr.home variable somewhere. In Windows, that was easy because
> you could just open up the system tray app for tomcat and set it
> there. How can I do that in linux? After googling, it suggested that I
> set an environment variable in the web.xml file inside the war file. I
> even saw a commented out section for setting the home variable. I set
> it in there but it still is giving me this error:
>
> INFO: looking for solr.xml: /usr/share/tomcat5/solr/solr.xml
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader 
> INFO: Solr home set to 'solr/'
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-6.1.3.jar' to
> Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/servlet-api-2.5-6.1.3.jar'
> to Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jetty-util-6.1.3.jar'
> to Solr classloader
> Aug 14, 2009 6:16:41 PM org.apache.solr.core.SolrResourceLoader
> createClassLoader
> INFO: Adding 'file:/usr/share/tomcat5/solr/lib/jsp-2.1/' to Solr
> classloader
> Aug 14, 2009 6:16:42 PM org.apache.solr.servlet.SolrDispatchFilter init
> SEVERE: Could not start SOLR. Check solr/home property
> java.lang.ExceptionInInitializerError
>
> Any ideas?
>
> --Aaron
>
> On Sat, Aug 15, 2009 at 12:40 AM, Shalin Shekhar
> Mangar wrote:
> > On Sat, Aug 15, 2009 at 6:52 AM, Aaron Aberg 
> wrote:
> >
> >> My client is using a dedicated server with plesk for the control
> >> panel. The hosting provider says that anything done using the control
> >> panel is supported by their tech support, so if i try anything using
> >> SSH, it voids that warranty. Its easy to install a servlet through
> >> plesk anyway, I upload the war file using the the java servlet
> >> installer. A sample servlet has been installed so I know this part
> >> works.
> >>
> >> However, when I install solr, i get what looks like a warning icon and
> >> if I hover over it the tool tip text says this:
> >>
> >> "Actual status of the application does not correspond to the status
> >> retrieved from the database."
> >>
> >> The host providers support team says that there is something wrong
> >> with the war file (big help). Since I kind of stuck using tomcat 5.5,
> >> is there an older version of solr that I should be using? How can I
> >> fix this so that I can use solr?
> >>
> >
> > Solr works fine with Tomcat 5.5
> >
> > Solr is a little different from other web applications in that it uses a
> > specified directory as "solr home" which contains configuration (schema,
> > solrconfig etc). The solrconfig also tells solr where to store the index.
> >
> > The solr home can be specified in many ways (environment variable, JNDI
> or
> > even in web.xml). I guess you should specify the absolute path to the
> solr
> > home in one of these ways (specifying it in web.xml and re-packaging the
> war
> > may be easier for you?). You'd need to upload the contents of the solr
> home
> > at the specified location on your server (look at the example directory
> > which comes with the solr download). Specify the absolute path to data
> > directory in solrconfig.xml (make sure it is writable).
> >
> > http://wiki.apache.org/solr/SolrTomcat
> > http://wiki.apache.org/solr/HackingSolr
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>

Re: Choosing between t and s field types

[Constantijn Visinescu]

I just checked and the default schema.xml for SOLR 1.3
(solr/conf/schema.xml.original) and i don't see tint, etc listed.
So either they''re new in 1.4 and I don't know about them or they were
manually defined.

Can you post your schema.xml entries for tint? (along with any comments it
might have?)

Constantijn

On Fri, Aug 14, 2009 at 1:39 PM, Ninad Raut wrote:

> Hi Constantijn,
> What are the t types viz;tint,tfloat etc. for? Is there a special use of
> these?
>
> On Fri, Aug 14, 2009 at 4:37 PM, Constantijn Visinescu
> wrote:
>
> > Accoridng to the documentation in schema.xml.original sint etc can be
> used
> > for both sorting and range queries?
> >
> >
> > > sortMissingLast="true" omitNorms="true"/>
> >
> >
> > On Fri, Aug 14, 2009 at 11:08 AM, Ninad Raut  > >wrote:
> >
> > > Hi,
> > > I want certain fields of type int,float and date to be sortable and I
> > > should
> > > be able to run my range queries as well as facet queries on those
> fields.
> > > Now as far as I know sint,sfloat fieldtypes  make the fields sortable
> and
> > > tint,tfloat,tdate allow range queries on the fields.
> > > I want both these features in my fields. How can I make this happen?
> > >
> >
>

Re: Choosing between t and s field types

[Constantijn Visinescu]

Accoridng to the documentation in schema.xml.original sint etc can be used
for both sorting and range queries?





On Fri, Aug 14, 2009 at 11:08 AM, Ninad Raut wrote:

> Hi,
> I want certain fields of type int,float and date to be sortable and I
> should
> be able to run my range queries as well as facet queries on those fields.
> Now as far as I know sint,sfloat fieldtypes  make the fields sortable and
> tint,tfloat,tdate allow range queries on the fields.
> I want both these features in my fields. How can I make this happen?
>

Re: Searching for reservations/availability with Solr

[Constantijn Visinescu]

Ok, i'm going to go and implement this solution then. Thanks for the help :)

On Wed, Aug 12, 2009 at 2:11 PM, Avlesh Singh  wrote:

> >
> >  > multiValued="true" />
> >
> I think this should be good enough with the "number of days since ..." kind
> of data.
>
> Cheers
> Avlesh
>
> On Wed, Aug 12, 2009 at 5:26 PM, Constantijn Visinescu
> wrote:
>
> > Hmm .. I looked up the transformer class and that should work, thanks for
> > the tip :)
> >
> > Thinking about it a bit more, wouldn't it be easier to just add a field
> > like
> >
> >  > multiValued="true" />
> >
> > And use a function in the transformer to converts the day to int (number
> of
> > days since jan 1st 2000 or something).
> >
> > Then if i use the same date to int conversion in my front end that
> queries
> > solr I can ask for all documents minus the range of int-ified dates that
> > the
> > user wants.
> >
> > That way I just have 1 field I can query and I dont have to mess with the
> > date to figure out the potential field name.
> >
> > Solr should be able to query this easily for 10 mil documents? I'm
> > relatively new to using Solr and i don't have a feeling yet for what
> types
> > of single querys can bring Solr to it's knees ;) (so far i haven't found
> > anything that's actually useful that takes more then a second on my 10
> mil
> > documents test index, so so far I'm happy and impressed :))
> >
> >
> > On Wed, Aug 12, 2009 at 1:18 PM, Avlesh Singh  wrote:
> >
> > > >
> > > > I'd have to add the year because things might end up reserved far
> > ahead,
> > > or
> > > > they might be reserved again on the same day next year.
> > > >
> > > If that is the case, then yes you'll have to take year into account
> too.
> > >
> > > Wouldn't it be terribly inefficient when I have 10million+ documents in
> > my
> > > > index and i want to check all of em for availabilty over a 2 week
> > period?
> > > > That'd mean 14 extra parts in my query. I'm a bit worried that this
> > might
> > > > mess up efficiency a lot?
> > > >
> > > I don't see a reason as to why 14 Boolean clauses in a query should be
> > > considered a problem. Actually, it is too small a number to worry
> about.
> > > Moreover, you can use a negated range query, as Shalin proposed. You
> > should
> > > make the field of type sortable-int in that case.
> > >
> > > Any way i could get the functionality you're describing without having
> to
> > > > basically write my own data import handler implementation
> > >
> > > Worst come worst, you may need to write a transformer.
> > >
> > > Cheers
> > > Avlesh
> > >
> > > On Wed, Aug 12, 2009 at 4:38 PM, Constantijn Visinescu
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > This seems like a bit of an unconventional suggestion but it just
> might
> > > > work.
> > > >
> > > > I'd have to add the year because things might end up reserved far
> > ahead,
> > > or
> > > > they might be reserved again on the same day next year.
> > > > I do have 2 questions:
> > > >
> > > > 1) Wouldn't it be terribly inefficient when I have 10million+
> documents
> > > in
> > > > my index and i want to check all of em for availabilty over a 2 week
> > > > period?
> > > > That'd mean 14 extra parts in my query. I'm a bit worried that this
> > might
> > > > mess up efficiency a lot? (Yes, I have aprox 10 mil documents in my
> > index
> > > > ;))
> > > >
> > > > 2) I populate my index with a data import handler (
> > > > http://wiki.apache.org/solr/DataImportHandler ) from a table that
> has
> > > > lines
> > > > that contain a foreign key to the room number, a reserved_from date
> and
> > a
> > > > reserved_to date. Any way i could get the functionality you're
> > describing
> > > > without having to basically write my own data import handler
> > > > implementation?
> > > >
> > > > Constantijn Visinescu
> > > >
> > > > On Wed, Aug 12, 2009 at 10:49 AM, Avlesh Singh 
> > wrote:
> > > >
> > &g

Re: Searching for reservations/availability with Solr

[Constantijn Visinescu]

Hmm .. I looked up the transformer class and that should work, thanks for
the tip :)

Thinking about it a bit more, wouldn't it be easier to just add a field like



And use a function in the transformer to converts the day to int (number of
days since jan 1st 2000 or something).

Then if i use the same date to int conversion in my front end that queries
solr I can ask for all documents minus the range of int-ified dates that the
user wants.

That way I just have 1 field I can query and I dont have to mess with the
date to figure out the potential field name.

Solr should be able to query this easily for 10 mil documents? I'm
relatively new to using Solr and i don't have a feeling yet for what types
of single querys can bring Solr to it's knees ;) (so far i haven't found
anything that's actually useful that takes more then a second on my 10 mil
documents test index, so so far I'm happy and impressed :))


On Wed, Aug 12, 2009 at 1:18 PM, Avlesh Singh  wrote:

> >
> > I'd have to add the year because things might end up reserved far ahead,
> or
> > they might be reserved again on the same day next year.
> >
> If that is the case, then yes you'll have to take year into account too.
>
> Wouldn't it be terribly inefficient when I have 10million+ documents in my
> > index and i want to check all of em for availabilty over a 2 week period?
> > That'd mean 14 extra parts in my query. I'm a bit worried that this might
> > mess up efficiency a lot?
> >
> I don't see a reason as to why 14 Boolean clauses in a query should be
> considered a problem. Actually, it is too small a number to worry about.
> Moreover, you can use a negated range query, as Shalin proposed. You should
> make the field of type sortable-int in that case.
>
> Any way i could get the functionality you're describing without having to
> > basically write my own data import handler implementation
>
> Worst come worst, you may need to write a transformer.
>
> Cheers
> Avlesh
>
> On Wed, Aug 12, 2009 at 4:38 PM, Constantijn Visinescu
> wrote:
>
> > Hi,
> >
> > This seems like a bit of an unconventional suggestion but it just might
> > work.
> >
> > I'd have to add the year because things might end up reserved far ahead,
> or
> > they might be reserved again on the same day next year.
> > I do have 2 questions:
> >
> > 1) Wouldn't it be terribly inefficient when I have 10million+ documents
> in
> > my index and i want to check all of em for availabilty over a 2 week
> > period?
> > That'd mean 14 extra parts in my query. I'm a bit worried that this might
> > mess up efficiency a lot? (Yes, I have aprox 10 mil documents in my index
> > ;))
> >
> > 2) I populate my index with a data import handler (
> > http://wiki.apache.org/solr/DataImportHandler ) from a table that has
> > lines
> > that contain a foreign key to the room number, a reserved_from date and a
> > reserved_to date. Any way i could get the functionality you're describing
> > without having to basically write my own data import handler
> > implementation?
> >
> > Constantijn Visinescu
> >
> > On Wed, Aug 12, 2009 at 10:49 AM, Avlesh Singh  wrote:
> >
> > > >
> > > > Searches would be for documents (rooms) that don't have certain dates
> > in
> > > > their multi-valued fields for the a particular month.
> > > > E.g if you wanted to find out rooms available on 15th, 16th and 17th
> of
> > > > August, the query could be:
> > > > q=!(+reserved_dates_August:15 +reserved_dates_August:16
> > > > +reserved_dates_August:17)
> > > >
> > >
> > > I was too fast to suggest the query above. The dates should be
> > individually
> > > negated.
> > >
> > > Cheers
> > > Avlesh
> > >
> > > On Tue, Aug 11, 2009 at 9:39 PM, Avlesh Singh 
> wrote:
> > >
> > > > From what I understood, you need a day level granularity (i.e booked
> on
> > > > 15th, 16th and 17th of August) in your indexes. If this is true, then
> > why
> > > > even store a "date"? For your use case, I think this should suffice -
> > > >  > > > stored="true" multiValued="true"/>
> > > >
> > > > Each document will have values like these for this particular field -
> > > > reserved_dates_August => 15, 16, 19
> > > > reserved_dates_September => 1, 3
> > > > 
> > > >
> > > > Searches would

Re: Searching for reservations/availability with Solr

[Constantijn Visinescu]

On Tue, Aug 11, 2009 at 6:13 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Tue, Aug 11, 2009 at 7:08 PM, Constantijn Visinescu
> wrote:
>
> >
> > 
> > Room1
> > 2000-08-01T00:00:00Z
> > 2000-08-31T23:59:59Z
> > 
> > 
> > Room2
> > 2000-08-01T00:00:00Z
> > 2000-08-13T23:59:59Z
> > 2000-08-20T00:00:00Z
> > 2000-08-22T23:59:59Z
> > 
> >
> > Now i want to run a query that gives me all documents(rooms) that are
> > avaiable from aug 15th to aug 18th (should return Room2), or from aug
> 10th
> > to aug 15th (should return none) or from sept 1st to sept 5th (should
> > return
> > both).
> >
> > Is it possible to run queries like this in solr? (either with my current
> > schema setup, or a different one that accomplishes the same idea).
> >
>
> I'm not sure how you are creating the dynamic field's name. But if you know
> the field name, the query can be like this:
>
> q=*:*&fq=-reserved_from:[2008-08-15T00:00:00Z TO 2008-08-18T00:00:00Z]
>
> Note the minus at the start of the fq. The above query says give me all
> documents (q=*:*) excluding the ones in which reserved_from is between 15th
> to 18th August.
>
> --
> Regards,
> Shalin Shekhar Mangar.


It wolud be ideal if i can get this working, but the above query includes
room 1, which is reserved for all of august, in the search results when i
want all the rooms that are avaliable from aug 15th, to aug 18th.

The above query would be ideal if i can get it working, I'm just not sure
how :) i've done some experimenting but i can't seem to get the query quite
right

Re: Searching for reservations/availability with Solr

[Constantijn Visinescu]

Hi,

This seems like a bit of an unconventional suggestion but it just might
work.

I'd have to add the year because things might end up reserved far ahead, or
they might be reserved again on the same day next year.
I do have 2 questions:

1) Wouldn't it be terribly inefficient when I have 10million+ documents in
my index and i want to check all of em for availabilty over a 2 week period?
That'd mean 14 extra parts in my query. I'm a bit worried that this might
mess up efficiency a lot? (Yes, I have aprox 10 mil documents in my index
;))

2) I populate my index with a data import handler (
http://wiki.apache.org/solr/DataImportHandler ) from a table that has lines
that contain a foreign key to the room number, a reserved_from date and a
reserved_to date. Any way i could get the functionality you're describing
without having to basically write my own data import handler implementation?

Constantijn Visinescu

On Wed, Aug 12, 2009 at 10:49 AM, Avlesh Singh  wrote:

> >
> > Searches would be for documents (rooms) that don't have certain dates in
> > their multi-valued fields for the a particular month.
> > E.g if you wanted to find out rooms available on 15th, 16th and 17th of
> > August, the query could be:
> > q=!(+reserved_dates_August:15 +reserved_dates_August:16
> > +reserved_dates_August:17)
> >
>
> I was too fast to suggest the query above. The dates should be individually
> negated.
>
> Cheers
> Avlesh
>
> On Tue, Aug 11, 2009 at 9:39 PM, Avlesh Singh  wrote:
>
> > From what I understood, you need a day level granularity (i.e booked on
> > 15th, 16th and 17th of August) in your indexes. If this is true, then why
> > even store a "date"? For your use case, I think this should suffice -
> >  > stored="true" multiValued="true"/>
> >
> > Each document will have values like these for this particular field -
> > reserved_dates_August => 15, 16, 19
> > reserved_dates_September => 1, 3
> > 
> >
> > Searches would be for documents (rooms) that don't have certain dates in
> > their multi-valued fields for the a particular month.
> > E.g if you wanted to find out rooms available on 15th, 16th and 17th of
> > August, the query could be:
> > q=!(+reserved_dates_August:15 +reserved_dates_August:16
> > +reserved_dates_August:17)
> >
> > Hope this helps.
> >
> > Cheers
> > Avlesh
> >
> >
> > On Tue, Aug 11, 2009 at 7:08 PM, Constantijn Visinescu <
> baeli...@gmail.com
> > > wrote:
> >
> >> Hello,
> >>
> >> I have a problem i'm trying to solve where i want to check if objects
> are
> >> reserved or not. (by reservation i mean like making a reservation at a
> >> hotel, because you would like to stay there on certain dates).
> >>
> >> I have the following in my schema.xml
> >>
> >> 
> >>  >> stored="true"/>
> >>  >> stored="true"/>
> >>
> >> and the follwoing 2 documents in Solr
> >>
> >> 
> >> Room1
> >> 2000-08-01T00:00:00Z
> >> 2000-08-31T23:59:59Z
> >> 
> >> 
> >> Room2
> >> 2000-08-01T00:00:00Z
> >> 2000-08-13T23:59:59Z
> >> 2000-08-20T00:00:00Z
> >> 2000-08-22T23:59:59Z
> >> 
> >>
> >> Now i want to run a query that gives me all documents(rooms) that are
> >> avaiable from aug 15th to aug 18th (should return Room2), or from aug
> 10th
> >> to aug 15th (should return none) or from sept 1st to sept 5th (should
> >> return
> >> both).
> >>
> >> Is it possible to run queries like this in solr? (either with my current
> >> schema setup, or a different one that accomplishes the same idea).
> >>
> >> I'm at a loss as to how to formulate a solr query to get the data i
> want.
> >>
> >> Thanks in advance,
> >> Constantijn Visinescu
> >>
> >
> >
>

Searching for reservations/availability with Solr

[Constantijn Visinescu]

Hello,

I have a problem i'm trying to solve where i want to check if objects are
reserved or not. (by reservation i mean like making a reservation at a
hotel, because you would like to stay there on certain dates).

I have the following in my schema.xml





and the follwoing 2 documents in Solr


Room1
2000-08-01T00:00:00Z
2000-08-31T23:59:59Z


Room2
2000-08-01T00:00:00Z
2000-08-13T23:59:59Z
2000-08-20T00:00:00Z
2000-08-22T23:59:59Z


Now i want to run a query that gives me all documents(rooms) that are
avaiable from aug 15th to aug 18th (should return Room2), or from aug 10th
to aug 15th (should return none) or from sept 1st to sept 5th (should return
both).

Is it possible to run queries like this in solr? (either with my current
schema setup, or a different one that accomplishes the same idea).

I'm at a loss as to how to formulate a solr query to get the data i want.

Thanks in advance,
Constantijn Visinescu