: > 2) "contrib" code that runs as it's own process to crawl documents and
: > send them to a Solr server. (mybe it parses them, or maybe it relies on
: > the next item...)
:
: Do you know FAST? It uses a step-by-step approach ("pipeline") in which
: all of these tasks are done. Much of it is tuned in a easy web tool.
:
: The point I'm trying to make is that contrib code is nice, but a
: "complete package" with these possibilities could broaden Solr's appeal
: somewhat.in my limited experience, commercial applications tend to be "all in one solutions" not so much because it really adds value that they are "all in one" but because it helps with vendor lock in -- companies tend to want to give you a single monolithic product, because if they gave you lots of little products that tried to do just one thing very well, you might decide that one of their little products is crap, and write your own replacement for just that piece using a great open-source library you found .. and then you might realize that this *other* open-source library would make it really easy for you to replace this other little piece of their system and would be a lot more efficient ... etc. the point being that once they've got you using a monolithic application, it's a lot harder to stop using the whole thing all at once, then it would be for you to stop using 1 of N mini-applications they provide. open source projects on the other hand, tend to work well when they are composed of lots of little pieces -- because little pieces are easier to work on when you have a finite number of developers working in their spare time, because each developer can work on a few peices at a time, and those peices can be reviewed/used by other people even if the system as a whole isn't finished. I ramble about this to try and explain why Solr may not be what you would consider a "complete package" at the moment .... and why it may never reach the state you think would make it a "complete package" ... because there are a lot of people out there who don't need it to be -- it would be hard to be a full blown GUI configurable, web crawling, document detecting, customizable schema based application and still allow for people to use small pieces of it. To put it another way: it's a lot easier for people to put reusable components with clean APIs together in interesting ways, then it is for people to extract reusable components with clean APIs from a monolithic application. : Exactly, this sounds more like it. But if similar inputstreams can be : handled by Nutch, what's the point in using Solr at all? The http API's? : In other words, both Nutch and Solr seem to have functionality that : enterprises would want. But neither gives you the "total solution". if what you care about is extracting text from arbitrary documents, that's what Nutch does well -- it doesn't worry about trying to extract complex structure from the documents, so it can parse/index lots of document formats into the same index. Solr's goal is to let *you* define the index format, but that requires you defining what data goes into which fields as well, and that makes generic reusable document crawlers parsers harder to get "right" in a way that can work for anyone. -Hoss
