Hi Brian,
I had similar questions when I begun to try and evaluate Solr.
If you use Java and SolrJ you might find these useful:
- http://wiki.apache.org/solr/Solrj#Streaming_documents_for_an_update
-
http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html
I am also interested in knowing what is the best and more efficient way
to index a large number of documents.
Paolo
Wawok, Brian wrote:
Hello,
I am using SOLR for some proof of concept work, and was wondering if anyone has
some guidance on a best practice.
Background:
Nightly get a delivery of a few 1000 reports. Each report is between 1 and
500,000 pages.
For my proof of concept I am using a single 100,000 page report.
I want to see how fast I can make SOLR handle this single report, and then can
see how we can scale out to meet the total indexing demand (if needed).
Trial 1:
1) Set up a solr server on server A with the default settings. Added a few
new fields to index, including a full text index of the report.
2) Set up a simple Python script on serve B. It splits the report into
100,000 small documents, pulls out a few key fields to be sent along to index,
and uses a python implementation of curl to shove the documents into the server
(with 4 threads posting away).
3) After all 100,000 documents are posted, we post an index and let the
server index.
I was able to get this method to work, and it took around 340 seconds for the
posting, and 10 seconds for the indexing. I am not sure if that indexing speed
is a red hearing, and it was really doing a little bit of the indexing during
the posts, or what.
Regardless, it seems less than ideal to make 100,000 requests to the server to
index 100,000 documents. Does anyone have an idea for how to make this process
more efficient? Should I look into making an XML document with 100,000
documents enclosed? Or what will give me the best performance? Will this be
much better than what I am seeing with my post method? I am not against
writing a custom parser on the SOLR side, but if there is already a way in SOLR
to send many documents efficiently, that is better.
Thanks!
Brian Wawok
