On Tue, 2007-01-16 at 16:28 +0100, Eivind Hasle Amundsen wrote: > First: Please pardon the cross-post to solr-user for reference. I hope > to continue this thread in solr-dev. Please answer to solr-dev. > > > 1) more documentation (and posisbly some locking configuration options) on > > how you can use Solr to access an index generated by the nutch crawler (i > > think Thorsten has allready done this) or by Compass, or any other system > > that builds a Lucene index. > > Thorsten Scherler?
Hmm, I did the exact opposite. Let me explain you my use case. I am working on a part of a portal http://andaluciajunta.es. The new version of http://andaluciajunta.es/BOJA is this part. The current version is based on a proprietary CMS in a dynamic environment. The new development is using Apache Forrest to generate static html. Now coming to solr/nutch, you can find http://andaluciajunta.es/portal/aj-bojaBuscador/0,22815,,00.html the current search engine especially for the BOJA. This will be changed to a solr powered solution. Like I said I only doing one part of the portal and the main portal has a search engine as well. http://andaluciajunta.es/aj-sea-.html This search engine will be based on nutch in the next version. The special character is that this main portal search engine has to search against the solr BOJA based indexed. Meaning Nutch will have to search the solr index and not vice versa. What I did before we decided to go with solr is a simple test. I copied my solr index into a nutch instance and dispatched a couple of queries. The only thing that you need is to keep your solr schema as close as possible to the one nutch uses. For example nutch is using "content", "url" and "title" as default fields when returning the search result. If you do not have this fields in your solr schema then nutch will return null. > Is this code available anywhere? Like stated above it is a couple of lines in the solr schema: <field name="title" type="string" stored="true" ></field> <field name="content" type="text" indexed="true" stored="true" /> <field name="url" type="string" stored="true" ></field> Then you just need to point your nutch instance to this index for searching. The same is true (I guess) for solr searching a nutch index. You could use nutch to update the index, point solr to the index and it should work (if you have defined all field in the schema). > Sounds very > interesting to me. Maybe someone could ellaborate on the differences > between the indexes created by Nutch/Solr/Compass/etc., or point me in > the direction of an answer? > I am far from being an expert, but actually the only real difference I see is the usage of field names. All indexes could be searched with a raw lucene component (if they are based on the same lucene version) > > 2) "contrib" code that runs as it's own process to crawl documents and > > send them to a Solr server. (mybe it parses them, or maybe it relies on > > the next item...) > > Do you know FAST? It uses a step-by-step approach ("pipeline") in which > all of these tasks are done. Much of it is tuned in a easy web tool. > > The point I'm trying to make is that contrib code is nice, but a > "complete package" with these possibilities could broaden Solr's appeal > somewhat. Hmm, I think like Hoss on this, why do we want do the same work of nutch. If you need a crawler why not use the one from nutch and change some lines? I actually use Forrest as crawler when I generate the new sites, which will then push the content to the solr server via a plugin I developed: http://forrest.apache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.output.solr/ > > > 3) Stock "update" plugins that can each read a raw inputstreams of a some > > widely used file format (PDF, RDF, HTML, XML of any schema) and have > > configuration options telling them them what fields in the schema each > > part of their document type should go in. > > Exactly, this sounds more like it. But if similar inputstreams can be > handled by Nutch, what's the point in using Solr at all? The http API's? > In other words, both Nutch and Solr seem to have functionality that > enterprises would want. But neither gives you the "total solution". > Not sure. I am using solr because I did not had to develop three different nutch plugin to make it work. Further I have punctual updates where I push a certain set of documents to the server, so no need for a crawler. > Don't get it wrong, I don't want to bloat the products, even though it > would be nice to have a crossover solution which is easy to set up. > > The architecture could look something like this: > > Connector -> Parser -> DocProc -> (via schema) -> Index > > Possible connectors: JDBC, filesystem, crawler, manual feed > Possible parsers: PDF, whatever > Both connectors, parsers AND the document processors would be plugins. > The DocProcs would typically be adjusted for each enterprise' needs, so > that it fits with their schema.xml. > > Problem is; I haven't worked enough with Solr, Nutch, Lucene etc. to > really know all possibilities and limitations. But I do believe that the > outlined architecture would be flexible and answer many needs. Not sure. > So the > question is: > > What is Solr missing? Could parts of Nutch be used in Solr to achieve > this? How? Have I misunderstood completely? :) > Solr and nutch is based on lucene. Meaning you COULD reuse nearly everything. From my own experience (I am a Apache Lenya and Apache Forrest committer, both based in cocoon) I must say that interproject collaboration is very hard to archive. Anyway if you need a crawler but want use solr then see the crawling code of nutch and write a standalone crawler that will update the solr index. salu2 > Eivind
