Re: Does Solr support integration with the Compass framework?
On 1/15/07, Lukas Vlcek <[EMAIL PROTECTED]> wrote: Ryan, Could you be more specific on your statement? On 1/12/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > > I started using compass a few months back. It is an amazing system: > with almost no effort, it just works. BUT the showstopper (for me) > was that you could not easily update the index from multiple machines. > Compass lets you put the lucene indexes in SQL with JDBC, but this > felt wrong then i found solr, and it solves most things. > What exactly do you mean by "you could not easily update the index from multiple machines"? Consider a standard load balanced web setup with three machines: DB1 - running mysql WEB1 - webapp talking to DB1 WEB2 - webapp talking to DB1 ... In compass, the lucene index is stored on a disk - WEB1 writes its lucene index on WEB1. So for changes that WEB1 makes, WEB2 does not see them (without adding more logic) To solve this problem, compass is able to write its index into SQL. WEB1 & WEB2 can write the lucene index in DB1. But the performance is not great and it seems to be something people discourage (tho i have not tried it) Also, take a look at: http://forums.opensymphony.com/thread.jspa?messageID=100071 Could you describe you problem in more details (and possible workaround if you found any) please? workaround? I'm now using solr :) Otherwise, consider: * Try the JDBC store: http://www.opensymphony.com/compass/versions/1.1M3/html/core-connection.html#core-connection-jdbc * If you are ok with WEB1 & WEB2 being slightly out of sync for new content, you could us them normally and periodically call index() on the hibernate GPS device. This will synchronize whatever is stored in hibernate with the lucene index
Re: Does Solr support integration with the Compass framework?
Hi Marios, It can store the index in a database, but I wouldn't want to use that route myself. Here is a quick link to the docs which provides an over-view of the transactional features; http://www.opensymphony.com/compass/versions/1.1M3/html/core-searchengine.html#core-searchengine-transaction HTH, Graham Marios Skounakis wrote: Does compass store the lucene index in a database? If this is the case, it is fairly straightforward to understand how this happens. If the index is still in disk files how does it provide transactional semantics? Would you care to give a high-level overview? TIA Marios On 1/15/07, Graham O'Regan <[EMAIL PROTECTED]> wrote: compass provides a transaction manager for lucene indexes so you can incorporate an index update and database update in a single transaction or roll-back if either fails. thats why it would be interesting to see the two working together. Marios Skounakis wrote: > Hi all, > >> > >> >I am working on a hibernate-solr bridge that will behave like the >> >compass Hibernate3GpsDevice. It gets a callback from hibernate when >> >an object is stored, checks if it is 'SolrDocumentable' and sends it >> >to solr using the client library from: >> > http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) >> > >> >If your interested, i can send you my initial version... when i'm >> >further along, i'll try to post it to solr/client/java >> >> That would be great - we're also facing the same issue of rolling our >> own code to keep a Solr index in sync with a MySQL DB that we access >> via Hibernate. > > I wonder whether people who try to keep a Solr (or Lucene) index in > sync with a database are at all worried about index update failures. > > Propagating the update from the DB to the index is one thing, and > relatively easy to implement. But how do you handle failures to update > either the index or the DB since you cannot enforce transactional > semantics over both updates? Or do index update failures occur so > infrequently that you do not worry about it? > > Marios >
Re: Does Solr support integration with the Compass framework?
Ryan, Could you be more specific on your statement? On 1/12/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I started using compass a few months back. It is an amazing system: with almost no effort, it just works. BUT the showstopper (for me) was that you could not easily update the index from multiple machines. Compass lets you put the lucene indexes in SQL with JDBC, but this felt wrong then i found solr, and it solves most things. What exactly do you mean by "you could not easily update the index from multiple machines"? Could you describe you problem in more details (and possible workaround if you found any) please? Thanks, Lukas
Re: Does Solr support integration with the Compass framework?
Does compass store the lucene index in a database? If this is the case, it is fairly straightforward to understand how this happens. If the index is still in disk files how does it provide transactional semantics? Would you care to give a high-level overview? TIA Marios On 1/15/07, Graham O'Regan <[EMAIL PROTECTED]> wrote: compass provides a transaction manager for lucene indexes so you can incorporate an index update and database update in a single transaction or roll-back if either fails. thats why it would be interesting to see the two working together. Marios Skounakis wrote: > Hi all, > >> > >> >I am working on a hibernate-solr bridge that will behave like the >> >compass Hibernate3GpsDevice. It gets a callback from hibernate when >> >an object is stored, checks if it is 'SolrDocumentable' and sends it >> >to solr using the client library from: >> > http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) >> > >> >If your interested, i can send you my initial version... when i'm >> >further along, i'll try to post it to solr/client/java >> >> That would be great - we're also facing the same issue of rolling our >> own code to keep a Solr index in sync with a MySQL DB that we access >> via Hibernate. > > I wonder whether people who try to keep a Solr (or Lucene) index in > sync with a database are at all worried about index update failures. > > Propagating the update from the DB to the index is one thing, and > relatively easy to implement. But how do you handle failures to update > either the index or the DB since you cannot enforce transactional > semantics over both updates? Or do index update failures occur so > infrequently that you do not worry about it? > > Marios >
Re: Does Solr support integration with the Compass framework?
compass provides a transaction manager for lucene indexes so you can incorporate an index update and database update in a single transaction or roll-back if either fails. thats why it would be interesting to see the two working together. Marios Skounakis wrote: Hi all, > >I am working on a hibernate-solr bridge that will behave like the >compass Hibernate3GpsDevice. It gets a callback from hibernate when >an object is stored, checks if it is 'SolrDocumentable' and sends it >to solr using the client library from: > http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) > >If your interested, i can send you my initial version... when i'm >further along, i'll try to post it to solr/client/java That would be great - we're also facing the same issue of rolling our own code to keep a Solr index in sync with a MySQL DB that we access via Hibernate. I wonder whether people who try to keep a Solr (or Lucene) index in sync with a database are at all worried about index update failures. Propagating the update from the DB to the index is one thing, and relatively easy to implement. But how do you handle failures to update either the index or the DB since you cannot enforce transactional semantics over both updates? Or do index update failures occur so infrequently that you do not worry about it? Marios
Re: Does Solr support integration with the Compass framework?
Hi all, > >I am working on a hibernate-solr bridge that will behave like the >compass Hibernate3GpsDevice. It gets a callback from hibernate when >an object is stored, checks if it is 'SolrDocumentable' and sends it >to solr using the client library from: > http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) > >If your interested, i can send you my initial version... when i'm >further along, i'll try to post it to solr/client/java That would be great - we're also facing the same issue of rolling our own code to keep a Solr index in sync with a MySQL DB that we access via Hibernate. I wonder whether people who try to keep a Solr (or Lucene) index in sync with a database are at all worried about index update failures. Propagating the update from the DB to the index is one thing, and relatively easy to implement. But how do you handle failures to update either the index or the DB since you cannot enforce transactional semantics over both updates? Or do index update failures occur so infrequently that you do not worry about it? Marios
Re: Does Solr support integration with the Compass framework?
doesn't compass use multiple indexes? yes, compass creates an index for each 'type' it handles. I 'think' you could point solr to it as long as you know that limitation. I started using compass a few months back. It is an amazing system: with almost no effort, it just works. BUT the showstopper (for me) was that you could not easily update the index from multiple machines. Compass lets you put the lucene indexes in SQL with JDBC, but this felt wrong then i found solr, and it solves most things. I am working on a hibernate-solr bridge that will behave like the compass Hibernate3GpsDevice. It gets a callback from hibernate when an object is stored, checks if it is 'SolrDocumentable' and sends it to solr using the client library from: http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) If your interested, i can send you my initial version... when i'm further along, i'll try to post it to solr/client/java That would be great - we're also facing the same issue of rolling our own code to keep a Solr index in sync with a MySQL DB that we access via Hibernate. Thanks! -- Ken -- Ken Krugler Krugle, Inc. +1 530-210-6378 "Find Code, Find Answers"
Re: Does Solr support integration with the Compass framework?
doesn't compass use multiple indexes? yes, compass creates an index for each 'type' it handles. I 'think' you could point solr to it as long as you know that limitation. I started using compass a few months back. It is an amazing system: with almost no effort, it just works. BUT the showstopper (for me) was that you could not easily update the index from multiple machines. Compass lets you put the lucene indexes in SQL with JDBC, but this felt wrong then i found solr, and it solves most things. I am working on a hibernate-solr bridge that will behave like the compass Hibernate3GpsDevice. It gets a callback from hibernate when an object is stored, checks if it is 'SolrDocumentable' and sends it to solr using the client library from: http://issues.apache.org/jira/browse/SOLR-20 (solr-client.zip) If your interested, i can send you my initial version... when i'm further along, i'll try to post it to solr/client/java ryan
Re: Does Solr support integration with the Compass framework?
: doesn't compass use multiple indexes?
I don't relaly know anything about Compass, but if it's managing seperate
index partitions itself then yes it would certainly be hard to try and
point a Solr instance at it's index (since you'd just have ot pick one)
Solr was designed with teh assumption that it would *own* the index and be
bale to manage everything itself -- butthe master/slave model built in to
it makes it easy to configure a 'slave" whose "master" isn't actaully
another Solr index -- with the constraint that it still expects the index
to be a single FSDirectory.
The topic of using a MultiReader (whose primary usage is opening
multi-segment indexes) to search multiple standalone indexes in seperate
directories has come up on the lucene-java lists before ... i've never
tried it myself but if:
a) it really is that easy
b) there is a desire among the Solr user base for Solr to support
searching of indexes not created by Solr (ie: Compass)
...then perhaps we could add two new options:
1) a way to configure solr so that it knows it should treat the index as
read only (this might be as simple as not registering any UpdateHandlers)
2) a hook to register a factory Solr would use when it wants/needs to
open a new IndexReader ... the default implimentation being the current
code that opens an FSDIrectory in ${dataDir}/index ... but another more
complex implimentation could be configured to take in a list of
directories and open a MultiReader across all of them. (or even an
implimentation that used a pure RAMDirectory
-Hoss
Re: Does Solr support integration with the Compass framework?
doesn't compass use multiple indexes? have a read of the "direct lucene" box on http://www.opensymphony.com/compass/versions/1.1M3/html/introduction.html#i-use-lucene would that prevent the two being used together? i'd be interested in getting the two working together as well, it'd be great to have the compass api to create the indexes and use solr to expose them over http. Yonik Seeley wrote: One could do a very loose coupling by just pointing Solr at the index created by Compass, and send a commit command to solr whenever you want a new view of the index. -Yonik On 1/10/07, Jochen Franke <[EMAIL PROTECTED]> wrote: Currently I'm investigating different Lucene based search technologies. For the indexing of our object model my favorite is Compass because of the Object/Search Engine Mapping capabilities. At the same time Solr offers serveral nice features like faceted search and caching. Has anybody integrated or tried to integrate Solr with Compass already and can share experiences. Thanks, Jochen
Re: Does Solr support integration with the Compass framework?
doesn't compass use multiple indexes? have a read of the "direct lucene" box on http://www.opensymphony.com/compass/versions/1.1M3/html/introduction.html#i-use-lucene would that prevent the two being used together? i'd be interested in getting the two working together as well, it'd be great to have the compass api to create the indexes and use solr to expose them over http. Yonik Seeley wrote: One could do a very loose coupling by just pointing Solr at the index created by Compass, and send a commit command to solr whenever you want a new view of the index. -Yonik On 1/10/07, Jochen Franke <[EMAIL PROTECTED]> wrote: Currently I'm investigating different Lucene based search technologies. For the indexing of our object model my favorite is Compass because of the Object/Search Engine Mapping capabilities. At the same time Solr offers serveral nice features like faceted search and caching. Has anybody integrated or tried to integrate Solr with Compass already and can share experiences. Thanks, Jochen
Re: Does Solr support integration with the Compass framework?
One could do a very loose coupling by just pointing Solr at the index created by Compass, and send a commit command to solr whenever you want a new view of the index. -Yonik On 1/10/07, Jochen Franke <[EMAIL PROTECTED]> wrote: Currently I'm investigating different Lucene based search technologies. For the indexing of our object model my favorite is Compass because of the Object/Search Engine Mapping capabilities. At the same time Solr offers serveral nice features like faceted search and caching. Has anybody integrated or tried to integrate Solr with Compass already and can share experiences. Thanks, Jochen
Does Solr support integration with the Compass framework?
Currently I'm investigating different Lucene based search technologies. For the indexing of our object model my favorite is Compass because of the Object/Search Engine Mapping capabilities. At the same time Solr offers serveral nice features like faceted search and caching. Has anybody integrated or tried to integrate Solr with Compass already and can share experiences. Thanks, Jochen
