[
https://issues.apache.org/jira/browse/CASSANDRA-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212617#comment-13212617
]
Jeremy Hanna edited comment on CASSANDRA-3843 at 2/21/12 2:30 PM:
------------------------------------------------------------------
I did repairs on all the nodes and then compacts on all the nodes. Then I did a
pig job to simply count the number of rows in the column family. Again I think
the overall writes were reduced but there are writes going on. I need to turn
debug on and do the same test again. I did the compactions at 6:42 and the
range scans at 14:16:
{code}
-rw-r--r-- 1 root root 40106228511 Feb 21 06:42 account_snapshot-g-792-Data.db
-rw-r--r-- 1 root root 206884816 Feb 21 06:42 account_snapshot-g-792-Filter.db
-rw-r--r-- 1 root root 2913796038 Feb 21 06:42 account_snapshot-g-792-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 06:42
account_snapshot-g-792-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-793-Compacted
-rw-r--r-- 1 root root 287286 Feb 21 14:16 account_snapshot-g-793-Data.db
-rw-r--r-- 1 root root 976 Feb 21 14:16 account_snapshot-g-793-Filter.db
-rw-r--r-- 1 root root 20857 Feb 21 14:16 account_snapshot-g-793-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:16
account_snapshot-g-793-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-794-Compacted
-rw-r--r-- 1 root root 87770771 Feb 21 14:17 account_snapshot-g-794-Data.db
-rw-r--r-- 1 root root 293944 Feb 21 14:17 account_snapshot-g-794-Filter.db
-rw-r--r-- 1 root root 6377968 Feb 21 14:17 account_snapshot-g-794-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-794-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-795-Compacted
-rw-r--r-- 1 root root 78459166 Feb 21 14:17 account_snapshot-g-795-Data.db
-rw-r--r-- 1 root root 262600 Feb 21 14:17 account_snapshot-g-795-Filter.db
-rw-r--r-- 1 root root 5698156 Feb 21 14:17 account_snapshot-g-795-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-795-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-796-Compacted
-rw-r--r-- 1 root root 69838937 Feb 21 14:17 account_snapshot-g-796-Data.db
-rw-r--r-- 1 root root 234000 Feb 21 14:17 account_snapshot-g-796-Filter.db
-rw-r--r-- 1 root root 5077447 Feb 21 14:17 account_snapshot-g-796-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-796-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-797-Compacted
-rw-r--r-- 1 root root 68094433 Feb 21 14:17 account_snapshot-g-797-Data.db
-rw-r--r-- 1 root root 227808 Feb 21 14:17 account_snapshot-g-797-Filter.db
-rw-r--r-- 1 root root 4943098 Feb 21 14:17 account_snapshot-g-797-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-797-Statistics.db
-rw-r--r-- 1 root root 304163307 Feb 21 14:20 account_snapshot-g-798-Data.db
-rw-r--r-- 1 root root 1019776 Feb 21 14:20 account_snapshot-g-798-Filter.db
-rw-r--r-- 1 root root 22096669 Feb 21 14:20 account_snapshot-g-798-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-798-Statistics.db
-rw-r--r-- 1 root root 65874829 Feb 21 14:18 account_snapshot-g-799-Data.db
-rw-r--r-- 1 root root 220192 Feb 21 14:18 account_snapshot-g-799-Filter.db
-rw-r--r-- 1 root root 4777809 Feb 21 14:18 account_snapshot-g-799-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-799-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-800-Compacted
-rw-r--r-- 1 root root 50067413 Feb 21 14:18 account_snapshot-g-800-Data.db
-rw-r--r-- 1 root root 167416 Feb 21 14:18 account_snapshot-g-800-Filter.db
-rw-r--r-- 1 root root 3632313 Feb 21 14:18 account_snapshot-g-800-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-800-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-801-Compacted
-rw-r--r-- 1 root root 50575719 Feb 21 14:18 account_snapshot-g-801-Data.db
-rw-r--r-- 1 root root 169160 Feb 21 14:18 account_snapshot-g-801-Filter.db
-rw-r--r-- 1 root root 3669880 Feb 21 14:18 account_snapshot-g-801-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-801-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-802-Compacted
-rw-r--r-- 1 root root 41788766 Feb 21 14:19 account_snapshot-g-802-Data.db
-rw-r--r-- 1 root root 139776 Feb 21 14:19 account_snapshot-g-802-Filter.db
-rw-r--r-- 1 root root 3033069 Feb 21 14:19 account_snapshot-g-802-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-802-Statistics.db
-rw-r--r-- 1 root root 46547146 Feb 21 14:19 account_snapshot-g-803-Data.db
-rw-r--r-- 1 root root 155720 Feb 21 14:19 account_snapshot-g-803-Filter.db
-rw-r--r-- 1 root root 3378457 Feb 21 14:19 account_snapshot-g-803-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-803-Statistics.db
-rw-r--r-- 1 root root 142719184 Feb 21 14:20 account_snapshot-g-804-Data.db
-rw-r--r-- 1 root root 478576 Feb 21 14:19 account_snapshot-g-804-Filter.db
-rw-r--r-- 1 root root 10356119 Feb 21 14:20 account_snapshot-g-804-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-804-Statistics.db
-rw-r--r-- 1 root root 55373874 Feb 21 14:19 account_snapshot-g-805-Data.db
-rw-r--r-- 1 root root 185160 Feb 21 14:19 account_snapshot-g-805-Filter.db
-rw-r--r-- 1 root root 4017391 Feb 21 14:19 account_snapshot-g-805-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-805-Statistics.db
-rw-r--r-- 1 root root 46399227 Feb 21 14:19 account_snapshot-g-806-Data.db
-rw-r--r-- 1 root root 155120 Feb 21 14:19 account_snapshot-g-806-Filter.db
-rw-r--r-- 1 root root 3365947 Feb 21 14:19 account_snapshot-g-806-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-806-Statistics.db
-rw-r--r-- 1 root root 58491393 Feb 21 14:19 account_snapshot-g-807-Data.db
-rw-r--r-- 1 root root 196048 Feb 21 14:19 account_snapshot-g-807-Filter.db
-rw-r--r-- 1 root root 4253922 Feb 21 14:19 account_snapshot-g-807-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-807-Statistics.db
-rw-r--r-- 1 root root 47609635 Feb 21 14:20 account_snapshot-g-808-Data.db
-rw-r--r-- 1 root root 159320 Feb 21 14:20 account_snapshot-g-808-Filter.db
-rw-r--r-- 1 root root 3456985 Feb 21 14:20 account_snapshot-g-808-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-808-Statistics.db
-rw-r--r-- 1 root root 46923060 Feb 21 14:20
account_snapshot-tmp-g-809-Data.db
-rw-r--r-- 1 root root 0 Feb 21 14:20
account_snapshot-tmp-g-809-Index.db
-rw-r--r-- 1 root root 49693602 Feb 21 14:20
account_snapshot-tmp-g-810-Data.db
-rw-r--r-- 1 root root 166600 Feb 21 14:20
account_snapshot-tmp-g-810-Filter.db
-rw-r--r-- 1 root root 3614750 Feb 21 14:20
account_snapshot-tmp-g-810-Index.db
{code}
was (Author: jeromatron):
I did repairs on all the nodes and then compacts on all the nodes. Then I
did a pig job to simply count the number of rows in the column family. Again I
think the overall writes were reduced but there are writes going on. I need to
turn debug on and do the same test again. I did the compactions at 6:42 and the
range scans at 14:16:
-rw-r--r-- 1 root root 40106228511 Feb 21 06:42 account_snapshot-g-792-Data.db
-rw-r--r-- 1 root root 206884816 Feb 21 06:42 account_snapshot-g-792-Filter.db
-rw-r--r-- 1 root root 2913796038 Feb 21 06:42 account_snapshot-g-792-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 06:42
account_snapshot-g-792-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-793-Compacted
-rw-r--r-- 1 root root 287286 Feb 21 14:16 account_snapshot-g-793-Data.db
-rw-r--r-- 1 root root 976 Feb 21 14:16 account_snapshot-g-793-Filter.db
-rw-r--r-- 1 root root 20857 Feb 21 14:16 account_snapshot-g-793-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:16
account_snapshot-g-793-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-794-Compacted
-rw-r--r-- 1 root root 87770771 Feb 21 14:17 account_snapshot-g-794-Data.db
-rw-r--r-- 1 root root 293944 Feb 21 14:17 account_snapshot-g-794-Filter.db
-rw-r--r-- 1 root root 6377968 Feb 21 14:17 account_snapshot-g-794-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-794-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-795-Compacted
-rw-r--r-- 1 root root 78459166 Feb 21 14:17 account_snapshot-g-795-Data.db
-rw-r--r-- 1 root root 262600 Feb 21 14:17 account_snapshot-g-795-Filter.db
-rw-r--r-- 1 root root 5698156 Feb 21 14:17 account_snapshot-g-795-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-795-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-796-Compacted
-rw-r--r-- 1 root root 69838937 Feb 21 14:17 account_snapshot-g-796-Data.db
-rw-r--r-- 1 root root 234000 Feb 21 14:17 account_snapshot-g-796-Filter.db
-rw-r--r-- 1 root root 5077447 Feb 21 14:17 account_snapshot-g-796-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-796-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-797-Compacted
-rw-r--r-- 1 root root 68094433 Feb 21 14:17 account_snapshot-g-797-Data.db
-rw-r--r-- 1 root root 227808 Feb 21 14:17 account_snapshot-g-797-Filter.db
-rw-r--r-- 1 root root 4943098 Feb 21 14:17 account_snapshot-g-797-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:17
account_snapshot-g-797-Statistics.db
-rw-r--r-- 1 root root 304163307 Feb 21 14:20 account_snapshot-g-798-Data.db
-rw-r--r-- 1 root root 1019776 Feb 21 14:20 account_snapshot-g-798-Filter.db
-rw-r--r-- 1 root root 22096669 Feb 21 14:20 account_snapshot-g-798-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-798-Statistics.db
-rw-r--r-- 1 root root 65874829 Feb 21 14:18 account_snapshot-g-799-Data.db
-rw-r--r-- 1 root root 220192 Feb 21 14:18 account_snapshot-g-799-Filter.db
-rw-r--r-- 1 root root 4777809 Feb 21 14:18 account_snapshot-g-799-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-799-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-800-Compacted
-rw-r--r-- 1 root root 50067413 Feb 21 14:18 account_snapshot-g-800-Data.db
-rw-r--r-- 1 root root 167416 Feb 21 14:18 account_snapshot-g-800-Filter.db
-rw-r--r-- 1 root root 3632313 Feb 21 14:18 account_snapshot-g-800-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-800-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-801-Compacted
-rw-r--r-- 1 root root 50575719 Feb 21 14:18 account_snapshot-g-801-Data.db
-rw-r--r-- 1 root root 169160 Feb 21 14:18 account_snapshot-g-801-Filter.db
-rw-r--r-- 1 root root 3669880 Feb 21 14:18 account_snapshot-g-801-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:18
account_snapshot-g-801-Statistics.db
-rw-r--r-- 1 root root 0 Feb 21 14:20 account_snapshot-g-802-Compacted
-rw-r--r-- 1 root root 41788766 Feb 21 14:19 account_snapshot-g-802-Data.db
-rw-r--r-- 1 root root 139776 Feb 21 14:19 account_snapshot-g-802-Filter.db
-rw-r--r-- 1 root root 3033069 Feb 21 14:19 account_snapshot-g-802-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-802-Statistics.db
-rw-r--r-- 1 root root 46547146 Feb 21 14:19 account_snapshot-g-803-Data.db
-rw-r--r-- 1 root root 155720 Feb 21 14:19 account_snapshot-g-803-Filter.db
-rw-r--r-- 1 root root 3378457 Feb 21 14:19 account_snapshot-g-803-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-803-Statistics.db
-rw-r--r-- 1 root root 142719184 Feb 21 14:20 account_snapshot-g-804-Data.db
-rw-r--r-- 1 root root 478576 Feb 21 14:19 account_snapshot-g-804-Filter.db
-rw-r--r-- 1 root root 10356119 Feb 21 14:20 account_snapshot-g-804-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-804-Statistics.db
-rw-r--r-- 1 root root 55373874 Feb 21 14:19 account_snapshot-g-805-Data.db
-rw-r--r-- 1 root root 185160 Feb 21 14:19 account_snapshot-g-805-Filter.db
-rw-r--r-- 1 root root 4017391 Feb 21 14:19 account_snapshot-g-805-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-805-Statistics.db
-rw-r--r-- 1 root root 46399227 Feb 21 14:19 account_snapshot-g-806-Data.db
-rw-r--r-- 1 root root 155120 Feb 21 14:19 account_snapshot-g-806-Filter.db
-rw-r--r-- 1 root root 3365947 Feb 21 14:19 account_snapshot-g-806-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-806-Statistics.db
-rw-r--r-- 1 root root 58491393 Feb 21 14:19 account_snapshot-g-807-Data.db
-rw-r--r-- 1 root root 196048 Feb 21 14:19 account_snapshot-g-807-Filter.db
-rw-r--r-- 1 root root 4253922 Feb 21 14:19 account_snapshot-g-807-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:19
account_snapshot-g-807-Statistics.db
-rw-r--r-- 1 root root 47609635 Feb 21 14:20 account_snapshot-g-808-Data.db
-rw-r--r-- 1 root root 159320 Feb 21 14:20 account_snapshot-g-808-Filter.db
-rw-r--r-- 1 root root 3456985 Feb 21 14:20 account_snapshot-g-808-Index.db
-rw-r--r-- 1 root root 4276 Feb 21 14:20
account_snapshot-g-808-Statistics.db
-rw-r--r-- 1 root root 46923060 Feb 21 14:20
account_snapshot-tmp-g-809-Data.db
-rw-r--r-- 1 root root 0 Feb 21 14:20
account_snapshot-tmp-g-809-Index.db
-rw-r--r-- 1 root root 49693602 Feb 21 14:20
account_snapshot-tmp-g-810-Data.db
-rw-r--r-- 1 root root 166600 Feb 21 14:20
account_snapshot-tmp-g-810-Filter.db
-rw-r--r-- 1 root root 3614750 Feb 21 14:20
account_snapshot-tmp-g-810-Index.db
> Unnecessary ReadRepair request during RangeScan
> ------------------------------------------------
>
> Key: CASSANDRA-3843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3843
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Philip Andronov
> Assignee: Jonathan Ellis
> Fix For: 1.0.8
>
> Attachments: 3843-v2.txt, 3843.txt
>
>
> During reading with Quorum level and replication factor greater then 2,
> Cassandra sends at least one ReadRepair, even if there is no need to do that.
> With the fact that read requests await until ReadRepair will finish it slows
> down requsts a lot, up to the Timeout :(
> It seems that the problem has been introduced by the CASSANDRA-2494,
> unfortunately I have no enought knowledge of Cassandra internals to fix the
> problem and do not broke CASSANDRA-2494 functionality, so my report without a
> patch.
> Code explanations:
> {code:title=RangeSliceResponseResolver.java|borderStyle=solid}
> class RangeSliceResponseResolver {
> // ....
> private class Reducer extends
> MergeIterator.Reducer<Pair<Row,InetAddress>, Row>
> {
> // ....
> protected Row getReduced()
> {
> ColumnFamily resolved = versions.size() > 1
> ?
> RowRepairResolver.resolveSuperset(versions)
> : versions.get(0);
> if (versions.size() < sources.size())
> {
> for (InetAddress source : sources)
> {
> if (!versionSources.contains(source))
> {
>
> // [PA] Here we are adding null ColumnFamily.
> // later it will be compared with the "desired"
> // version and will give us "fake" difference which
> // forces Cassandra to send ReadRepair to a given
> source
> versions.add(null);
> versionSources.add(source);
> }
> }
> }
> // ....
> if (resolved != null)
>
> repairResults.addAll(RowRepairResolver.scheduleRepairs(resolved, table, key,
> versions, versionSources));
> // ....
> }
> }
> }
> {code}
> {code:title=RowRepairResolver.java|borderStyle=solid}
> public class RowRepairResolver extends AbstractRowResolver {
> // ....
> public static List<IAsyncResult> scheduleRepairs(ColumnFamily resolved,
> String table, DecoratedKey<?> key, List<ColumnFamily> versions,
> List<InetAddress> endpoints)
> {
> List<IAsyncResult> results = new
> ArrayList<IAsyncResult>(versions.size());
> for (int i = 0; i < versions.size(); i++)
> {
> // On some iteration we have to compare null and resolved which
> are obviously
> // not equals, so it will fire a ReadRequest, however it is not
> needed here
> ColumnFamily diffCf = ColumnFamily.diff(versions.get(i),
> resolved);
> if (diffCf == null)
> continue;
> // ....
> {code}
> Imagine the following situation:
> NodeA has X.1 // row X with the version 1
> NodeB has X.2
> NodeC has X.? // Unknown version, but because write was with Quorum it is 1
> or 2
> During the Quorum read from nodes A and B, Cassandra creates version 12 and
> send ReadRepair, so now nodes has the following content:
> NodeA has X.12
> NodeB has X.12
> which is correct, however Cassandra also will fire ReadRepair to NodeC. There
> is no need to do that, the next consistent read have a chance to be served by
> nodes {A, B} (no ReadRepair) or by pair {?, C} and in that case ReadRepair
> will be fired and brings nodeC to the consistent state
> Right now we are reading from the Index a lot and starting from some point in
> time we are getting TimeOutException because cluster is overloaded by the
> ReadRepairRequests *even* if all nodes has the same data :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
