Hi,
I am creating an index of approx 500K documents. I wrote an indexing
program using embeded solr: http://wiki.apache.org/solr/EmbeddedSolr
and am seeing probably a 10 fold increase in indexing speeds. My
problem is though, that if I try to reindex say 20K docs at a time it
slows down considerably. I currently batch my updates in lots of 100
and between batches I close and reopen the "connection" to solr like so:
private void openConnection(String environment) throws
ParserConfigurationException, IOException, SAXException {
System.setProperty("solr.solr.home", SOLR_HOME);
solrConfig = new SolrConfig("solrconfig.xml");
solrCore = new SolrCore(SOLR_HOME + "data/" + environment,
solrConfig, new IndexSchema(solrConfig, "schema.xml"));
logger.debug("Opened solr connection");
}
private void closeConnection() {
solrCore.close();
solrCore = null;
logger.debug("Closed solr connection");
}
Does anyone have any pointers or see anything obvious I'm doing wrong?
Thanks
PS Sorry if this is posted twice.