[
https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832570#comment-13832570
]
Lyuben Todorov edited comment on CASSANDRA-5351 at 11/26/13 1:27 PM:
---------------------------------------------------------------------
After [this
commit|https://github.com/lyubent/cassandra/commit/903e416539cdde78514850bda25076f3f2fc57ec]
to keep unrepaired data at L0 repairs start failing to validate after a few
inserts and compactions. The stack trace from the error in each node is below
(3 node ccm cluster was used here with the repair being issued to node 2).
{code}
INFO 15:19:10,321 Starting repair command #3, repairing 2 ranges for keyspace
test
INFO 15:19:10,322 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] new session:
will sync /127.0.0.2, /127.0.0.3 on range
(-9223372036854775808,-3074457345618258603] for test.[lvl]
INFO 15:19:10,325 Handshaking version with /127.0.0.3
INFO 15:19:10,343 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] requesting
merkle trees for lvl (to [/127.0.0.3, /127.0.0.2])
INFO 15:19:11,493 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] Received
merkle tree for lvl from /127.0.0.3
ERROR 15:19:16,138 Failed creating a merkle tree for [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]], /127.0.0.2 (see log for details)
ERROR 15:19:16,138 Exception in thread Thread[ValidationExecutor:2,1,main]
java.lang.AssertionError: row DecoratedKey(-9223264645216044815,
73636c744c546e56534c4741775141) received out of order wrt
DecoratedKey(-3331959603918038206, 685863786a586464616b794f597075)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:820)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
ERROR 15:19:16,139 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] session
completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO 15:19:16,138 Range (3074457345618258602,-9223372036854775808] has already
been repaired. Skipping repair.
S.AD: lvl repairedAt: 1385466263703792000
ERROR 15:19:16,139 Exception in thread Thread[AntiEntropySessions:5,5,RMI
Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
... 3 more
INFO 15:19:16,139 No column family to repair for keyspace test
ERROR 15:19:16,140 Repair session 55f4d610-569d-11e3-b553-975f903ccf5a for
range (-9223372036854775808,-3074457345618258603] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2401)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
... 3 more
{code}
{code}
// NODE 1 LOG
empty.
{code}
{code}
// NODE 3 LOG
INFO 15:19:10,351 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-67
(41127 bytes)
INFO 15:19:10,354 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-66
(1049680 bytes)
INFO 15:19:10,358 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-51
(1093563 bytes)
INFO 15:19:10,362 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-55
(1093563 bytes)
INFO 15:19:10,366 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-47
(1093563 bytes)
INFO 15:19:10,372 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-63
(1093563 bytes)
INFO 15:19:10,376 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-59
(1093563 bytes)
INFO 15:19:10,381 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-53
(1093563 bytes)
INFO 15:19:10,386 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-45
(1093563 bytes)
INFO 15:19:10,390 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-57
(1093563 bytes)
INFO 15:19:10,394 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-49
(1093563 bytes)
INFO 15:19:10,399 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-61
(1093563 bytes)
INFO 15:19:10,403 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-56
(1090759 bytes)
INFO 15:19:11,460 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] Sending
completed merkle tree to /127.0.0.2 for test/lvl
INFO 15:19:11,461 Handshaking version with /127.0.0.2
INFO 15:19:16,139 Starting anticompaction for 1 ranges.
INFO 15:19:16,145 Skipping anticompaction for 1, required sstable was
compacted and is no longer available.
INFO 15:19:16,145 Completed anticompaction successfully
{code}
was (Author: lyubent):
After [this
commit|https://github.com/lyubent/cassandra/commit/903e416539cdde78514850bda25076f3f2fc57ec]
to keep unrepaired data at L0 repairs start failing to validate after a few
inserts and compactions. The stack trace from the error in each node is below
(3 node ccm cluster was used here with the repair being issued to node 2).
{code}
INFO 15:19:10,321 Starting repair command #3, repairing 2 ranges for keyspace
test
INFO 15:19:10,322 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] new session:
will sync /127.0.0.2, /127.0.0.3 on range
(-9223372036854775808,-3074457345618258603] for test.[lvl]
INFO 15:19:10,325 Handshaking version with /127.0.0.3
INFO 15:19:10,343 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] requesting
merkle trees for lvl (to [/127.0.0.3, /127.0.0.2])
INFO 15:19:11,493 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] Received
merkle tree for lvl from /127.0.0.3
ERROR 15:19:16,138 Failed creating a merkle tree for [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]], /127.0.0.2 (see log for details)
ERROR 15:19:16,138 Exception in thread Thread[ValidationExecutor:2,1,main]
java.lang.AssertionError: row DecoratedKey(-9223264645216044815,
73636c744c546e56534c4741775141) received out of order wrt
DecoratedKey(-3331959603918038206, 685863786a586464616b794f597075)
at org.apache.cassandra.repair.Validator.add(Validator.java:136)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:820)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
ERROR 15:19:16,139 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] session
completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO 15:19:16,138 Range (3074457345618258602,-9223372036854775808] has already
been repaired. Skipping repair.
S.AD: lvl repairedAt: 1385466263703792000
ERROR 15:19:16,139 Exception in thread Thread[AntiEntropySessions:5,5,RMI
Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
... 3 more
INFO 15:19:16,139 No column family to repair for keyspace test
ERROR 15:19:16,140 Repair session 55f4d610-569d-11e3-b553-975f903ccf5a for
range (-9223372036854775808,-3074457345618258603] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2401)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#55f4d610-569d-11e3-b553-975f903ccf5a on test/lvl,
(-9223372036854775808,-3074457345618258603]] Validation failed in /127.0.0.2
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:212)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:91)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
... 3 more
{/code}
{code}
// NODE 1 LOG
empty.
{/code}
{code}
// NODE 3 LOG
INFO 15:19:10,351 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-67
(41127 bytes)
INFO 15:19:10,354 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-66
(1049680 bytes)
INFO 15:19:10,358 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-51
(1093563 bytes)
INFO 15:19:10,362 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-55
(1093563 bytes)
INFO 15:19:10,366 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-47
(1093563 bytes)
INFO 15:19:10,372 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-63
(1093563 bytes)
INFO 15:19:10,376 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-59
(1093563 bytes)
INFO 15:19:10,381 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-53
(1093563 bytes)
INFO 15:19:10,386 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-45
(1093563 bytes)
INFO 15:19:10,390 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-57
(1093563 bytes)
INFO 15:19:10,394 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-49
(1093563 bytes)
INFO 15:19:10,399 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-61
(1093563 bytes)
INFO 15:19:10,403 Opening
/Users/me/.ccm/5351/node3/data/test/lvl/snapshots/55f4d610-569d-11e3-b553-975f903ccf5a/test-lvl-jb-56
(1090759 bytes)
INFO 15:19:11,460 [repair #55f4d610-569d-11e3-b553-975f903ccf5a] Sending
completed merkle tree to /127.0.0.2 for test/lvl
INFO 15:19:11,461 Handshaking version with /127.0.0.2
INFO 15:19:16,139 Starting anticompaction for 1 ranges.
INFO 15:19:16,145 Skipping anticompaction for 1, required sstable was
compacted and is no longer available.
INFO 15:19:16,145 Completed anticompaction successfully
{/code}
> Avoid repairing already-repaired data by default
> ------------------------------------------------
>
> Key: CASSANDRA-5351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5351
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Lyuben Todorov
> Labels: repair
> Fix For: 2.1
>
>
> Repair has always built its merkle tree from all the data in a columnfamily,
> which is guaranteed to work but is inefficient.
> We can improve this by remembering which sstables have already been
> successfully repaired, and only repairing sstables new since the last repair.
> (This automatically makes CASSANDRA-3362 much less of a problem too.)
> The tricky part is, compaction will (if not taught otherwise) mix repaired
> data together with non-repaired. So we should segregate unrepaired sstables
> from the repaired ones.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
