This may be a bit off topic: How do you index an existing website and control the data going into index?
We already have Java code to process the HTML (or XHTML) and turn it into a SolrJ Document (removing tags and other things we do not want in the index). We use SolrJ for indexing. So I guess the question is essentially which Java crawler could be useful. We used to use wget on command line in our publishing process, but we do no longer want to do that. Thanks, Alexander
