With the release of Nutch 1.0 I think it is a good time to begin a
discussion about the future of Nutch. Here are some things to consider
and would love to here everyones views on this
Nutch's original intention was as a large-scale www search engine. That
is a very specific goal. Only a few people and organizations actually
use it on that level. (I just happen to be one of them as most of my
work focuses on large scale web search as opposed to vertical search).
Many, perhaps most, people using Nutch these days are either using parts
of Nutch, such as the crawler, or are targeting towards vertical or
intranet type search engines. This can be seen in how many people have
already started using the Solr integration features. So while Nutch was
originally intended as a www search, IMO most people aren't using it for
that purpose.
Since there are different purposes for different users, would it be good
to consider moving Nutch to a top level apache project out from under
the Lucene umbrella? This would then allow the creation of nutch
sub-projects, such as nutch-solr, nutch-hbase. Thoughts?
Many parts of Nutch have also been implemented in other projects. For
example, Tika for the parsers, Droids for the Crawler. In begs the
question what is Nutch's core features going forward. When I think
about search (again my perspective is large scale), I think crawling or
acquisition of data, parsing, analysis, indexing, deployment, and
searching. I personally think that there is much room for improvement
in crawling and especially analysis. Nutch shouldn't just be about the
shell but also the brains.
And one of the biggest things I see is many newcomers to nutch have a
very hard time getting started. Part of this is understanding mapreduce
mentality, part is documentation, part is there is only so much time
some of us have to answer questions so some questions go unanswered on
the lists. How might this be improved going forward?
Any other thoughts also welcome. Really I want to start a discussion
about where everyone thinks we are with the state of Nutch and its future.
Dennis
- The Future of Nutch Dennis Kubes
-
