On May 12, 2006, at 1:02 PM, Michael Levy wrote:
One nice feature of INQUERY is that you can create one large SGML file, containing lots of records, each bracketed with <DOC> and </ DOC> tags. Submitting that big SGML document for indexing goes very fast. I believe that Solr indexes one document at a time; each document requires a separate HTTP POST.
Actually adding multiple documents per POST is possible
How efficient is making a separate HTTP request per-document, when there are millions of documents? Do people ever use Solr's or Lucene's API directly for indexing large numbers of documents, and if so, what are the considerations pro and con?
Maybe Solr could evolve a facility for doing these types of bulk operations without HTTP, but still using Solr's engine somehow via API directly. I guess this gets tricky when you have a live Solr system up and juggling write locks though.
But currently going through HTTP is the only way, and likely to not be that much of a bottleneck especially given you can post multiple documents at a time (the wiki has an example, but I can't get to the web at the moment to post the link).
Erik