[
https://issues.apache.org/jira/browse/CASSANDRA-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241829#comment-16241829
]
Ludovic Boutros commented on CASSANDRA-13403:
---------------------------------------------
[~ifesdjeen],
in order to reproduce, I'm using a small C* 3.10 cluster with at least 4 nodes
and 256 tokens.
Here is my keyspace declaration :
{code:sql}
CREATE KEYSPACE lubo_test WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3'} AND durable_writes = true;
CREATE TABLE lubo_test.t_doc (
id text,
r int,
cid timeuuid,
PRIMARY KEY (id, r, cid)
) WITH CLUSTERING ORDER BY (r DESC, cid DESC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE CUSTOM INDEX i_doc ON lubo_test.t_doc (r) USING
'org.apache.cassandra.index.sasi.SASIIndex';
{code}
I'have added some docs :
{code:sql}
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '1', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '2', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '3', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '4', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '5', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '6', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '7', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '8', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '9', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '10', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '11', 0, now());
{code}
Then this query without repair :
{code:sql}
cassandra@cqlsh> SELECT * from lubo_test.t_doc where r = 0;
id | r | cid
----+---+--------------------------------------
6 | 0 | 66f68be0-c316-11e7-a464-03e2ed27ae86
7 | 0 | 66f74f30-c316-11e7-a464-03e2ed27ae86
9 | 0 | 66f97210-c316-11e7-a464-03e2ed27ae86
10 | 0 | 66faaa90-c316-11e7-a464-03e2ed27ae86
4 | 0 | 66f46900-c316-11e7-a464-03e2ed27ae86
3 | 0 | 66f37ea0-c316-11e7-a464-03e2ed27ae86
5 | 0 | 66f5a180-c316-11e7-a464-03e2ed27ae86
8 | 0 | 66f83990-c316-11e7-a464-03e2ed27ae86
2 | 0 | 66f29440-c316-11e7-a464-03e2ed27ae86
11 | 0 | 66fb6de0-c316-11e7-a464-03e2ed27ae86
1 | 0 | 66ea56e0-c316-11e7-a464-03e2ed27ae86
(11 rows)
{code}
If I fire a "nodetool repair --full", then I have :
{code:sql}
cassandra@cqlsh> SELECT * from lubo_test.t_doc where r = 0;
id | r | cid
----+---+--------------------------------------
6 | 0 | 66f68be0-c316-11e7-a464-03e2ed27ae86
7 | 0 | 66f74f30-c316-11e7-a464-03e2ed27ae86
10 | 0 | 66faaa90-c316-11e7-a464-03e2ed27ae86
4 | 0 | 66f46900-c316-11e7-a464-03e2ed27ae86
3 | 0 | 66f37ea0-c316-11e7-a464-03e2ed27ae86
5 | 0 | 66f5a180-c316-11e7-a464-03e2ed27ae86
2 | 0 | 66f29440-c316-11e7-a464-03e2ed27ae86
1 | 0 | 66ea56e0-c316-11e7-a464-03e2ed27ae86
(8 rows)
{code}
I can fire a "rebuild_index" on each node to fix the problem.
I've checked the debug log differences between 3 and 4 nodes.
It seems that with 3 nodes, the Anticompaction process is not done. You can see
in the log "mutating repairedAt instead of anticompacting".
Currently, I would bet that when anticompacting, the SASI Index is not rebuilt
correctly, but that's just a bet.
I've attached the log extractions, if you need more, just ask.
> nodetool repair breaks SASI index
> ---------------------------------
>
> Key: CASSANDRA-13403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13403
> Project: Cassandra
> Issue Type: Bug
> Components: sasi
> Environment: 3.10
> Reporter: Igor Novgorodov
> Assignee: Alex Petrov
>
> I've got table:
> {code}
> CREATE TABLE cservice.bulks_recipients (
> recipient text,
> bulk_id uuid,
> datetime_final timestamp,
> datetime_sent timestamp,
> request_id uuid,
> status int,
> PRIMARY KEY (recipient, bulk_id)
> ) WITH CLUSTERING ORDER BY (bulk_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX bulk_recipients_bulk_id ON cservice.bulks_recipients
> (bulk_id) USING 'org.apache.cassandra.index.sasi.SASIIndex';
> {code}
> There are 11 rows in it:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> Let's query by index (all rows have the same *bulk_id*):
> {code}
> > select * from bulks_recipients where bulk_id =
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> >
> ...
> (11 rows)
> {code}
> Ok, everything is fine.
> Now i'm doing *nodetool repair --partitioner-range --job-threads 4 --full* on
> each node in cluster sequentially.
> After it finished:
> {code}
> > select * from bulks_recipients where bulk_id =
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> ...
> (2 rows)
> {code}
> Only two rows.
> While the rows are actually there:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> If i issue an incremental repair on a random node, i can get like 7 rows
> after index query.
> Dropping index and recreating it fixes the issue. Is it a bug or am i doing
> the repair the wrong way?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
