[jira] [Commented] (CASSANDRA-17273) Lazy transaction log replica creation allows incorrect replica content divergence during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483442#comment-17483442 ] Caleb Rackliffe commented on CASSANDRA-17273: - +1 to all PRs (left a minor nit on the 3.0 version, which looks like it applies to the others as well) > Lazy transaction log replica creation allows incorrect replica content > divergence during compaction > --- > > Key: CASSANDRA-17273 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17273 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair, Local/Compaction >Reporter: Caleb Rackliffe >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x > > Time Spent: 20m > Remaining Estimate: 0h > > Recently encountered this around compaction/anticompaction: > {noformat} > 2022-01-13 10:18:24,325 ERROR [main] > org.apache.cassandra.db.lifecycle.LogTransaction - Unexpected disk state: > failed to read transaction log > [mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log in > .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f, > .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f] > Files and contents follow: > .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168] > > REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485] > > REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924] > COMMIT:[,0,0][2613697770] > .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350437-big,0,8][4051162457] > ***Does not match > > in first replica file > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168] > > REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485] > > REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924] > COMMIT:[,0,0][2613697770] > {noformat} > We have two data directories and two transaction log files, but one is > missing an ADD entry when the contents of the two log replicas should be > identical. One scenario that can cause this is the following: > 1. Start anticompaction on a single file, in directory {{/tmp/d0}}. > 2. Call {{trackNew()}} with 2 new files, both in a single directory, but in > directory {{/tmp/d1}}. This initializes the log file in {{/tmp/d1}}, but > there is still no log file in {{/tmp/d0}}. > 3. Anticompaction only writes to one of the files in {{/tmp/d1}} (say all > other keys were outside the repaired range). > 4. When anticompaction is done, the empty writer is aborted and we call > {{untrackNew()}}, which removes the added file from the registered log > “records" (BUT NOT FROM DISK in {{/tmp/d1}}). > 5. The REMOVE record is added. This references {{/tmp/d0}}. We lazily create > the log file there by dumping all the records we have in memory to that file, > which does not include the aborted SSTable above. > 6. Now the log files contain: > {noformat} > /tmp/d1/logfile.log: > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-2-big,0,8][3268492367] > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425] > REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379] > COMMIT:[,0,0][2613697770] > ** /tmp/d0/logfile.log: > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425] > REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379] > COMMIT:[,0,0][2613697770] > {noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.
[jira] [Commented] (CASSANDRA-17273) Lazy transaction log replica creation allows incorrect replica content divergence during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479921#comment-17479921 ] Marcus Eriksson commented on CASSANDRA-17273: - It is not as likely to occur on 3.11+, but still needs to be fixed (compactions write to multiple directories after range movements for example) > Lazy transaction log replica creation allows incorrect replica content > divergence during compaction > --- > > Key: CASSANDRA-17273 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17273 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair, Local/Compaction >Reporter: Caleb Rackliffe >Assignee: Caleb Rackliffe >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x > > > Recently encountered this around compaction/anticompaction: > {noformat} > 2022-01-13 10:18:24,325 ERROR [main] > org.apache.cassandra.db.lifecycle.LogTransaction - Unexpected disk state: > failed to read transaction log > [mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log in > .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f, > .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f] > Files and contents follow: > .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168] > > REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485] > > REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924] > COMMIT:[,0,0][2613697770] > .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350437-big,0,8][4051162457] > ***Does not match > > in first replica file > > ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168] > > REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485] > > REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924] > COMMIT:[,0,0][2613697770] > {noformat} > We have two data directories and two transaction log files, but one is > missing an ADD entry when the contents of the two log replicas should be > identical. One scenario that can cause this is the following: > 1. Start anticompaction on a single file, in directory {{/tmp/d0}}. > 2. Call {{trackNew()}} with 2 new files, both in a single directory, but in > directory {{/tmp/d1}}. This initializes the log file in {{/tmp/d1}}, but > there is still no log file in {{/tmp/d0}}. > 3. Anticompaction only writes to one of the files in {{/tmp/d1}} (say all > other keys were outside the repaired range). > 4. When anticompaction is done, the empty writer is aborted and we call > {{untrackNew()}}, which removes the added file from the registered log > “records" (BUT NOT FROM DISK in {{/tmp/d1}}). > 5. The REMOVE record is added. This references {{/tmp/d0}}. We lazily create > the log file there by dumping all the records we have in memory to that file, > which does not include the aborted SSTable above. > 6. Now the log files contain: > {noformat} > /tmp/d1/logfile.log: > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-2-big,0,8][3268492367] > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425] > REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379] > COMMIT:[,0,0][2613697770] > ** /tmp/d0/logfile.log: > ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425] > REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379] > COMMIT:[,0,0][2613697770] > {noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org