[
https://issues.apache.org/jira/browse/SOLR-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660457#action_12660457
]
noble.paul edited comment on SOLR-906 at 1/3/09 2:10 AM:
---------------------------------------------------------
* One problem with the current implementation is that it writes everything to
a local buffer and then uploads the whole content in one go. So essentially we
are wasting time till your 40K docs are written into this huge XML. Another
issue is that this XML has to fit in memory. We need to fix the
comonsHttpSolrServer first. It must stream the docs .
* We can enhance the SolrServer API by adding a method
SolrServer#add(Iterator<SolrInputDocs> docs) . So CommonsHttpSolrServer can
start writing the documents as and when you are producing your documents . We
also have the advantage of not storing the huge list of docs in memory.
* Another enhancement is using a different format (SOLR-865). It uses javabin
format and it can be extremely fast compared to XML and the payload can be
reduced substantially.
Probably we can overcome the perf problems to a certain extent with these two
fixes.
was (Author: noble.paul):
One problem with the current implementation is that it writes everything to
a local buffer and then uploads the whole content in one go. So essentially we
are wasting time till your 40K docs are written into this huge XML. Another
issue is that this XML has to fit in memory
We need to fix the comonsHttpSolrServer first. It must stream the docs .
Another enhancement is using a different format (SOLR-865). It uses javabin
format and it can be extremely fast compared to XML and the payload can be
reduced substantially.
Probably we can overcome the perf problems to a certain extent with these two
fixes.
> Buffered / Streaming SolrServer implementaion
> ---------------------------------------------
>
> Key: SOLR-906
> URL: https://issues.apache.org/jira/browse/SOLR-906
> Project: Solr
> Issue Type: New Feature
> Components: clients - java
> Reporter: Ryan McKinley
> Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: SOLR-906-StreamingHttpSolrServer.patch,
> SOLR-906-StreamingHttpSolrServer.patch,
> SOLR-906-StreamingHttpSolrServer.patch,
> SOLR-906-StreamingHttpSolrServer.patch, StreamingHttpSolrServer.java
>
>
> While indexing lots of documents, the CommonsHttpSolrServer add(
> SolrInputDocument ) is less then optimal. This makes a new request for each
> document.
> With a "StreamingHttpSolrServer", documents are buffered and then written to
> a single open Http connection.
> For related discussion see:
> http://www.nabble.com/solr-performance-tt9055437.html#a20833680
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
