On 3/3/2016 11:36 PM, sangs8788 wrote:
> When a commit fails, the document doesnt get cleared out from MQ and there is
> a task which runs in a background to republish the files to SOLR. If we do a
> batch commit we will not know we will end up redoing the same batch commit
> again. We currenlty have a client side commit which issue the command to
> SOLR. commit() returns a status code. If we are planning to use
> commitwithin(), I dont think it will actually return any result from solr
> since it is time oriented.
Do your indexing and commits in batches, as already recommended. I'd
start with 1000 and go up or down from there as needed. If the batch
indexing fails, or the commit fails, consider the entire batch failed.
That may not be the end of the world, though -- if the indexing was
successful (and didn't use ConcurrentUpdateSolrClient), then those
updates will be stored in the Solr transaction log, and will be replayed
if Solr is restarted or the core is reloaded.
If you want to be absolutely certain that the update/commit succeeded by
verifying data, one thing you *could* do is send a batch update, do a
commit, and then request every document in the batch with a query that
includes a limited fl parameter, and verify that the document is present
and the values of the fields requested in the fl parameter are correct.
I would probably do that query with {!cache=false} to avoid polluting
Solr's caches.
Almost every index update you make can be simply made again without
danger. The exceptions are certain kinds of atomic updates, and certain
situations with deletes. It's probably best to avoid doing those kinds
of updates, which are described below:
If you're doing atomic updates that increment or decrement a field
value, or atomic updates that add a new value to a multivalued field,
the results will be wrong if that update is repeated, although they
would be correct if the update is replayed from Solr's transaction log,
because atomic updates are no longer atomic when they hit the
transaction log -- they include values for every field in the document,
as if the document were built from scratch.
If you are explicitly deleting a document before replacing it, and those
actions were re-done in the opposite order, then the document would be
missing from the index. Because Solr handles the deletion automatically
when a document is being updated/replaced, explicit deleting is not
recommended for those situations.
> If we go with SOLR autocommit, is there a way to send a response to MQ
> saying commit successful ?
If commits are completely automatic (autoCommit, autoSoftCommit, or
commitWithin), there's no way for a program to be sure that they have
completed.
The general recommendation for Solr indexing, especially if your
pipeline is multi-threaded, is to simply send your updates, let Solr
handle commits, and rely on the design of Lucene combined with Solr's
transaction logs to keep your data safe. This approach does mean that
when things go wrong it may be a while before new data is searchable.
Emir's reply is spot on. Solr is not recommended as a primary data store.
Thanks,
Shawn
