Tim Owen created SOLR-9918:
------------------------------
Summary: An UpdateRequestProcessor to skip duplicate inserts and
ignore updates to missing docs
Key: SOLR-9918
URL: https://issues.apache.org/jira/browse/SOLR-9918
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: update
Reporter: Tim Owen
This is an UpdateRequestProcessor and Factory that we have been using in
production, to handle 2 common cases that were awkward to achieve using the
existing update pipeline and current processor classes:
* When inserting document(s), if some already exist then quietly skip the new
document inserts - do not churn the index by replacing the existing documents
and do not throw a noisy exception that breaks the batch of inserts. By analogy
with SQL, {{insert if not exists}}. In our use-case, multiple application
instances can (rarely) process the same input so it's easier for us to de-dupe
these at Solr insert time than to funnel them into a global ordered queue first.
* When applying AtomicUpdate documents, if a document being updated does not
exist, quietly do nothing - do not create a new partially-populated document
and do not throw a noisy exception about missing required fields. By analogy
with SQL, {{update where id = ..}}. Our use-case relies on this because we
apply updates optimistically and have best-effort knowledge about what
documents will exist, so it's easiest to skip the updates (in the same way a
Database would).
I would have kept this in our own package hierarchy but it relies on some
package-scoped methods, and seems like it could be useful to others if they
choose to configure it. Some bits of the code were borrowed from
{{DocBasedVersionConstraintsProcessorFactory}}.
Attached patch has unit tests to confirm the behaviour.
This class can be used by configuring solrconfig.xml like so..
{noformat}
<updateRequestProcessorChain name="skipexisting">
<processor class="solr.LogUpdateProcessorFactory" />
<processor
class="org.apache.solr.update.processor.SkipExistingDocumentsProcessorFactory">
<bool name="skipInsertIfExists">true</bool>
<bool name="skipUpdateIfMissing">false</bool> <!-- We will override this
per-request -->
</processor>
<processor class="solr.DistributedUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
{noformat}
and initParams defaults of
{noformat}
<str name="update.chain">skipexisting</str>
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
