On May 5, 2011, at 4:15 AM, Simon Willnauer wrote: > Hey folks > > On Tue, May 3, 2011 at 6:49 PM, Michael McCandless > <luc...@mikemccandless.com> wrote: >> Isn't our end goal here a bunch of well factored search modules? Ie, >> fast forward a year or two and I think we should have modules like >> these: > > I think we have two camps here (10k feet view): >
I'd say 3 camps: > 1. wants to move towards modularization might support all the modules > mike has listed below > 2. wants to stick with Solr's current architecture and remain > "monolithic" (not negative in this case) as much as possible 3. Those who think most should be modularized, but realize it's a ton of work for an unproven gain (although most admit it is a highly likely gain) and should be handled on a case-by-case basis as people do the work. I don't have anything against modularization, I just know, given my schedule, I won't be able to block off weeks of time to do it. I'm happy to review where/when I can. > > I think we can meet somewhere in between and agree on certain module > that should be available to lucene users as well. The ones I have in > mind are > primary search features like: > - Faceting Yeah, for instance, Bobo seems to have some interesting faceting implementations that are ASL, perhaps we can combine into this new faceting module. > - Highlighting > - Suggest > - Function Query (consolidation is needed here!) > - Analyzer factories +1. > > things like distribution and replication should remain in solr IMO but > might be moved to a more extensible API so that people can add their > own implementation. And, of course, all the web tier stuff (response writers, inputs, etc.) > I am thinking about things like the ZooKeeper > support that might not be a good solution for everybody where folks > have already JGroups infrastructure. Or other similar solutions. I wonder about using a ZeroConf implementation that can do self-discovery. > So I think we can work towards 2 > distinct goals. > 1. extract common search features into modules > 2. refactor solr to be more "elastic" / "distributed" and extensible > with respect to those goals. 3. Make it easier for Solr to be programmatically configured by decoupling the reading of schema.xml and solrconfig.xml from the code that actually contains the structures for the properties (IndexSchema and SolrConfig) > > maybe we can get agreement on such a basis though. > > let me know what you think I think it's reasonable. At the end of the day, it broadens the appeal of both Lucene and Solr. Solr still exists and is not just a "shell" and at the end of the day, remains the primary choice for people who don't want to stitch everything together themselves. All of it is easier to contribute to b/c people can focus in on the core area they know w/o having to know everything else per se. Stuff should be better tested b/c of it as well since it will receive broader use. That being said, and not to be discouraging, but I see it as a ton of work. > > simon >> >> * Faceting >> >> * Highlighting >> >> * Suggest (good patch is on LUCENE-2995) >> >> * Schema >> >> * Query impls >> >> * Query parsers >> >> * Analyzers (good progress here already, thanks Robert!), >> incl. factories/XML configuration (still need this) >> >> * Database import (DIH) >> >> * Web app >> >> * Distribution/replication >> >> * Doc set representations >> >> * Collapse/grouping >> >> * Caches >> >> * Similarity/scoring impls (BM25, etc.) >> >> * Codecs >> >> * Joins >> >> * Lucene core >> >> In this future, much of this code came from what is now Solr and >> Lucene, but we should freely and aggressively poach from other >> projects when appropriate (and license/provenance is OK). >> >> I keep seeing all these cool "compressed int set" projects popping >> up... surely these are useful for us. Solr poached a doc set impl >> from Nutch; probably there's other stuff to poach from Nutch, Mahout, >> etc. >> >> Katta's doing something sweet with distribution/replication; let's >> poach & merge w/ Solr's approach. There are various facet impls out >> there (Bobo browse/Zoie; Toke's; Elastic Search); let's poach & merge >> with Solr's. >> >> Elastic Search has lots of cool stuff, too, under ASL2. >> >> All these external open-source projects are fair game for poaching and >> refactoring into shared modules, along with what is now Solr and >> Lucene sources. >> >> In this ideal future, Solr becomes the bundling and default/example >> configuration of the Web App and other modules, much like how the >> various Linux distros bundle different stuff together around the Linux >> kernel. And if you are an advanced app and don't need the webapp >> part, you can cherry pick the huper duper modules you do need and >> directly embedded into your app. >> >> Isn't this the future we are working towards? >> >> Mike >> >> http://blog.mikemccandless.com >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > -------------------------- Grant Ingersoll Lucene Revolution -- Lucene and Solr User Conference May 25-26 in San Francisco www.lucenerevolution.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
