Just a heads-up that we released version 0.2. This might be of interest to the Tika community, since it contains parsers for both robots.txt and sitemaps.
-- Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr
