[ https://issues.apache.org/jira/browse/SOLR-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824614#comment-15824614 ]
ASF subversion and git services commented on SOLR-9941: ------------------------------------------------------- Commit 302ce326d5a2eee043445918fa3e3885dc003b2f in lucene-solr's branch refs/heads/branch_6x from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=302ce32 ] SOLR-9941: Clear deletes lists before log replay > log replay redundently (pre-)applies DBQs as if they were out of order > ---------------------------------------------------------------------- > > Key: SOLR-9941 > URL: https://issues.apache.org/jira/browse/SOLR-9941 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Hoss Man > Assignee: Ishan Chattopadhyaya > Attachments: SOLR-9941.hoss-test-experiment.patch, SOLR-9941.patch, > SOLR-9941.patch, SOLR-9941.patch, SOLR-9941.patch, SOLR-9941.patch > > > There's kind of an odd situation that arises when a Solr node starts up > (after a crash) and tries to recover from it's tlog that causes deletes to be > redundantly & excessively applied -- at a minimum it causes confusing really > log messages.... > * {{UpdateLog.init(...)}} creates {{TransactionLog}} instances for the most > recent log files found (based on numRecordsToKeep) and then builds a > {{RecentUpdates}} instance from them > * Delete entries from the {{RecentUpdates}} are used to populate 2 lists: > ** {{deleteByQueries}} > ** {{oldDeletes}} (for deleteById). > * Then when {{UpdateLog.recoverFromLog}} is called a {{LogReplayer}} is used > to replay any (uncommited) {{TransactionLog}} enteries > ** during replay {{UpdateLog}} delegates to the UpdateRequestProcessorChain > to for the various adds/deletes, etc... > ** when an add makes it to {{RunUpdateProcessor}} it delegates to > {{DirectUpdateHandler2}}, which (independent of the fact that we're in log > replay) calls {{UpdateLog.getDBQNewer}} for every add, looking for any > "Reordered" deletes that have a version greater then the add > *** if it finds _any_ DBQs "newer" then the document being added, it does a > low level {{IndexWriter.updateDocument}} and then immediately executes _all_ > the newer DBQs ... _once per add_ > ** these deletes are *also* still executed as part of the normal tlog replay, > because they are in the tlog. > Which means if you are recovering from a tlog with 90 addDocs, followed by 5 > DBQs, then *each* of those 5 DBQs will each be executed 91 times -- and for > 90 of those executions, a DUH2 INFO log messages will say {{"Reordered DBQs > detected. ..."}} even tough the only reason they are out of order is because > Solr is deliberately applying them out of order. > * At a minimum we should improve the log messages > * Ideally we should stop (pre-emptively) applying these deletes during tlog > replay. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org