[
https://issues.apache.org/jira/browse/ZOOKEEPER-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408373#comment-13408373
]
Patrick Hunt commented on ZOOKEEPER-1489:
-----------------------------------------
That's great, thanks: https://reviews.apache.org/r/5815/
fwiw the test was/is complicated. I didn't want to change too much and lose the
ability to reproduce the problem. I did replace a number of timeouts and such,
might not have caught everything. I'm also fine to log everything is that's an
issue.
> Data loss after truncate on transaction log
> -------------------------------------------
>
> Key: ZOOKEEPER-1489
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1489
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.3, 3.3.5
> Environment: Tested on Ubuntu 12.04 and CentOS 6, should be
> reproducible elsewhere
> Reporter: Christian Ziech
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.3.6, 3.4.4, 3.5.0
>
> Attachments: TruncateTxLogCorruption.tgz,
> TruncateTxLogCorruption.tgz, ZOOKEEPER-1489.patch, ZOOKEEPER-1489_br33.patch,
> ZOOKEEPER-1489_br34.patch
>
>
> The truncate method on the transaction log in the class
> org.apache.zookeeper.server.persistence.FileTxnLog will reduce the file size
> to the required amount without either closing or re-positioning the logStream
> (which could also be dangerous since the truncate method is not synchronized
> against concurrent writes to the log).
> This causes the next append to that log to create a small "hole" in the file
> which java would interpret as binary zeroes when reading it. This then causes
> to the FileTxnIterator.next() implementation to detect the end of the log
> file too early.
> I'll attach a small maven project with one junit test which can be used to
> reproduce the issue. Due to the blackbox nature of the test it will run for
> roughly 50 seconds unfortunately.
> Steps to reproduce:
> - Start an ensemble of zookeeper servers with at least 3 participants
> - Create one entry and the remove one of the servers from the ensemble
> temporarily (e.g. zk-2)
> - Create another entry which is hence only reflected on zk-1 and zk-3
> - Take zk-1 out of the ensemble without shutting it down (that is important,
> I did that by interrupting the network connection to that node) and clean zk-3
> - Bring back zk-2 and zk-3 so that they form a quorum
> - Allow zk-1 to connect again
> - zk-1 will receive a TRUNC message from zk-2 since zk-1 is now a minority
> knowing about that second node creation event
> - Create a third node
> - Force zk-1 to become master somehow
> - That third node will be gone
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
