Hoss Man created SOLR-9941:
------------------------------
Summary: log replay redundently (pre-)applies DBQs as if they were
out of order
Key: SOLR-9941
URL: https://issues.apache.org/jira/browse/SOLR-9941
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Hoss Man
There's kind of an odd situation that arises when a Solr node starts up (after
a crash) and tries to recover from it's tlog that causes deletes to be
redundantly & excessively applied -- at a minimum it causes confusing really
log messages....
* {{UpdateLog.init(...)}} creates {{TransactionLog}} instances for the most
recent log files found (based on numRecordsToKeep) and then builds a
{{RecentUpdates}} instance from them
* Delete entries from the {{RecentUpdates}} are used to populate 2 lists:
** {{deleteByQueries}}
** {{oldDeletes}} (for deleteById).
* Then when {{UpdateLog.recoverFromLog}} is called a {{LogReplayer}} is used to
replay any (uncommited) {{TransactionLog}} enteries
** during replay {{UpdateLog}} delegates to the UpdateRequestProcessorChain to
for the various adds/deletes, etc...
** when an add makes it to {{RunUpdateProcessor}} it delegates to
{{DirectUpdateHandler2}}, which (independent of the fact that we're in log
replay) calls {{UpdateLog.getDBQNewer}} for every add, looking for any
"Reordered" deletes that have a version greater then the add
*** if it finds _any_ DBQs "newer" then the document being added, it does a low
level {{IndexWriter.updateDocument}} and then immediately executes _all_ the
newer DBQs ... _once per add_
** these deletes are *also* still executed as part of the normal tlog replay,
because they are in the tlog.
Which means if you are recovering from a tlog with 90 addDocs, followed by 5
DBQs, then *each* of those 5 DBQs will each be executed 91 times -- and for 90
of those executions, a DUH2 INFO log messages will say {{"Reordered DBQs
detected. ..."}} even tough the only reason they are out of order is because
Solr is deliberately applying them out of order.
* At a minimum we should improve the log messages
* Ideally we should stop (pre-emptively) applying these deletes during tlog
replay.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
