[ 
https://issues.apache.org/jira/browse/SOLR-8263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044802#comment-15044802
 ] 

Renaud Delbru commented on SOLR-8263:
-------------------------------------

While the patch SOLR-8263-trunk-3 which added the dedup logic for the buffered 
updates seems straightforward, it introduced an issue which could lead to loss 
of documents.
The dedup logic was using the version of the last operation from the tlog files 
transferred from the master as a starting point for the dedup logic. However, 
these tlog files were not in synch with the index commit point, there were 
likely ahead of the index commit point (i.e., there were containing operations 
that occurred after the index commit point). Therefore, the starting point of 
the dedup logic was ahead of the index commit point, and therefore it was 
dropping all operations that occurred between the index commit point and the 
time the tlog files were transferred from the master.
In order to solve this, we had to modify the ReplicationHandler to filter out 
tlog files that were not associated to a given commit point. To find the tlog 
files associated to an index commit point, we fetch the max version of an index 
commit using VersionInfo.getMaxVersionFromIndex and use this version number to 
discard tlog files. Tlog file name encodes the version of their starting 
operation (this was originally used for seeking more efficiently across 
multiple tlog files), and we use this starting version to discard tlog that 
were created after the commit point (i.e., if starting version > max version).
The new patch committed by Erick includes this approach.


> Tlog replication could interfere with the replay of buffered updates
> --------------------------------------------------------------------
>
>                 Key: SOLR-8263
>                 URL: https://issues.apache.org/jira/browse/SOLR-8263
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Renaud Delbru
>            Assignee: Erick Erickson
>             Fix For: 5.5, 6.0
>
>         Attachments: SOLR-6273-plus-8263-5x.patch, SOLR-8263-trunk-1.patch, 
> SOLR-8263-trunk-2.patch, SOLR-8263-trunk-3.patch, SOLR-8263.patch
>
>
> The current implementation of the tlog replication might interfere with the 
> replay of the buffered updates. The current tlog replication works as follow:
> 1) Fetch the the tlog files from the master
> 2) reset the update log before switching the tlog directory
> 3) switch the tlog directory and re-initialise the update log with the new 
> directory.
> Currently there is no logic to keep "buffered updates" while resetting and 
> reinitializing the update log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to