On Aug 29, 2008, at 2:03 PM, Yonik Seeley wrote:
I've been thinking about the next major version of Solr.
Here's some brainstorming on goals/ideas:
- use a standard IOC container for externalization of configuration
and plugins... Spring "springs" to mind as the obvious choice here.
May want to use other spring services such as JMX integration, and
look into JMX management (more than just statistics), etc.
- support programatic construction and manipulation of IndexSchema,
etc.
Definitely. I'm a firm believer we spend too much time on
configuration workarounds. Also, the IOC layer definitely makes the
second point here trivial.
It's maybe worthwhile to at least _consider_ being able to transform
1.x configurations to 2.x, but not saying we have to.
- support some sort of standard RPC mechanism (Thrift, Etch, ???) so
strongly typed language bindings don't have to be developed for every
language (as it seems most people want). Create an IDL for common
operations and then use a compiler to create the stubs for perl,
python, java, etc.
- an RPC mechanism that can have multiple operations pending per
socket (and maybe use NIO) would probably be good for distributed
search, etc.
- allow more lower level index operations... create a new index at a
given spot, merge multiple indicies, etc.
+1
- make Solr more scalable and cloud computing friendly... make it
easier to create and deploy clusters/shards, as well as change the
size of clusters
People are definitely pushing on the scale front, it will be good to
have it baked in from the ground up.
- remove the single-master points of failure per-shard (support
or incorporate something like bailey)
- make it easier to deploy config changes (possibly use
zookeeper... prob want that for cluster management anyway)
- since solr will have the data, possibly allow plugins that
could do map-reduce, or other interfaces that enable things like
mahout.
Ah, the marriage of Solr and Mahout. Words cannot express my joy.
Think automatic classification and named entity recognition over large
scale distributed collections, all faceted and categorized and tied
together in eternal bliss. Sigh. (Dang, that guy's weird!)
See also https://issues.apache.org/jira/browse/SOLR-651
I think it's also useful to think about how other NLP type tools plug
in (i.e. sentence/paragraph detection, POS taggers, clustering,
categorization, etc.) Solr, thanks to it's pluggable output and
SearchComponent/Req Handler architecture can actually play quite well
with these things.
- support more changes w/o manual re-indexing... change the schema
and have Solr re-index in the background (assuming all data is
available via stored fields or elsewhere via a plugin)
Cool
- support more "realtime" search... greatly reducing or eliminating
the lag between adding a document and making it searchable
One of the big things people often want
- support "tagging" type of updates... quickly updating part of a
document, or data associated with a document
- try to expose more lower-level Lucene functionality to better
support other projects that want to embed Solr (IOC should hopefully
make Solr easier to embed and customize too)
Yep.
To support some of these goals, some re-architecture is probably in
the cards. Caching based on the IndexReader rather than the
IndexSearcher is probably one necessary change. We should also use
this as an opportunity to clean some things up and improve the core
architecture since this will be a major version change. But we should
also
- continue to support the current main solr web interfaces for
searching and update
Definitely a huge win.
- retain (or improve) the ease of use factor
- we should always be able to point at an existing Lucene index
and do interesting things with it
Even w/o a schema?
- continue to focus on single-node ease of use for small web
developers
Yes, this is the majority of users, I would guess, i.e. sites in the
range of less than 10 million docs.
As for the future of Solr 1.x, I fully expect a Solr 1.4 release as
well as other 1.x releases after that.
Possible next steps:
- Have discussions on solr-dev with a subject prefix of "solr2:"
- We should avoid the temptation to start banging out code (unless
it's just example code) and take some time to really leverage all of
the architectural experience this larger solr-dev community brings.
- Establish a wiki section for solr2 to capture current consensus...
but generally use solr-dev for ideas and establishing that consensus
- let java-dev know about this (i.e. what in Solr didn't suit their
needs and how can we change that)
Onward and upward... Other thoughts & ideas?
Better support for Spans, Payloads, Term Vectors. Granted, Spans just
need support via a query parser and the results written to the output,
but Payloads are a bit trickier when it comes to the indexing side of
thing.
-Grant
