[ 
https://issues.apache.org/jira/browse/CASSANDRA-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241829#comment-16241829
 ] 

Ludovic Boutros commented on CASSANDRA-13403:
---------------------------------------------

[~ifesdjeen],

in order to reproduce, I'm using a small C* 3.10 cluster with at least 4 nodes 
and 256 tokens.

Here is my keyspace declaration :

{code:sql}
CREATE KEYSPACE lubo_test WITH replication = {'class': 
'NetworkTopologyStrategy', 'dc1': '3'}  AND durable_writes = true;

CREATE TABLE lubo_test.t_doc (
    id text,
    r int,
    cid timeuuid,
    PRIMARY KEY (id, r, cid)
) WITH CLUSTERING ORDER BY (r DESC, cid DESC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE CUSTOM INDEX i_doc ON lubo_test.t_doc (r) USING 
'org.apache.cassandra.index.sasi.SASIIndex';
{code}

I'have added some docs :

{code:sql}
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '1', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '2', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '3', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '4', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '5', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '6', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '7', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '8', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '9', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '10', 0, now());
INSERT INTO lubo_test.t_doc ( id , r , cid ) VALUES ( '11', 0, now());
{code}

Then this query without repair :

{code:sql}
cassandra@cqlsh> SELECT * from lubo_test.t_doc where r = 0;

 id | r | cid
----+---+--------------------------------------
  6 | 0 | 66f68be0-c316-11e7-a464-03e2ed27ae86
  7 | 0 | 66f74f30-c316-11e7-a464-03e2ed27ae86
  9 | 0 | 66f97210-c316-11e7-a464-03e2ed27ae86
 10 | 0 | 66faaa90-c316-11e7-a464-03e2ed27ae86
  4 | 0 | 66f46900-c316-11e7-a464-03e2ed27ae86
  3 | 0 | 66f37ea0-c316-11e7-a464-03e2ed27ae86
  5 | 0 | 66f5a180-c316-11e7-a464-03e2ed27ae86
  8 | 0 | 66f83990-c316-11e7-a464-03e2ed27ae86
  2 | 0 | 66f29440-c316-11e7-a464-03e2ed27ae86
 11 | 0 | 66fb6de0-c316-11e7-a464-03e2ed27ae86
  1 | 0 | 66ea56e0-c316-11e7-a464-03e2ed27ae86

(11 rows)
{code}

If I fire a "nodetool repair --full", then I have :

{code:sql}
cassandra@cqlsh> SELECT * from lubo_test.t_doc where r = 0;

 id | r | cid
----+---+--------------------------------------
  6 | 0 | 66f68be0-c316-11e7-a464-03e2ed27ae86
  7 | 0 | 66f74f30-c316-11e7-a464-03e2ed27ae86
 10 | 0 | 66faaa90-c316-11e7-a464-03e2ed27ae86
  4 | 0 | 66f46900-c316-11e7-a464-03e2ed27ae86
  3 | 0 | 66f37ea0-c316-11e7-a464-03e2ed27ae86
  5 | 0 | 66f5a180-c316-11e7-a464-03e2ed27ae86
  2 | 0 | 66f29440-c316-11e7-a464-03e2ed27ae86
  1 | 0 | 66ea56e0-c316-11e7-a464-03e2ed27ae86

(8 rows)
{code}

I can fire a "rebuild_index" on each node to fix the problem.

I've checked the debug log differences between 3 and 4 nodes.

It seems that with 3 nodes, the Anticompaction process is not done. You can see 
in the log "mutating repairedAt instead of anticompacting".
Currently, I would bet that when anticompacting, the SASI Index is not rebuilt 
correctly, but that's just a bet.

I've attached the log extractions, if you need more, just ask.

> nodetool repair breaks SASI index
> ---------------------------------
>
>                 Key: CASSANDRA-13403
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13403
>             Project: Cassandra
>          Issue Type: Bug
>          Components: sasi
>         Environment: 3.10
>            Reporter: Igor Novgorodov
>            Assignee: Alex Petrov
>
> I've got table:
> {code}
> CREATE TABLE cservice.bulks_recipients (
>     recipient text,
>     bulk_id uuid,
>     datetime_final timestamp,
>     datetime_sent timestamp,
>     request_id uuid,
>     status int,
>     PRIMARY KEY (recipient, bulk_id)
> ) WITH CLUSTERING ORDER BY (bulk_id ASC)
>     AND bloom_filter_fp_chance = 0.01
>     AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
>     AND comment = ''
>     AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
>     AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND crc_check_chance = 1.0
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 0
>     AND gc_grace_seconds = 864000
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX bulk_recipients_bulk_id ON cservice.bulks_recipients 
> (bulk_id) USING 'org.apache.cassandra.index.sasi.SASIIndex';
> {code}
> There are 11 rows in it:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> Let's query by index (all rows have the same *bulk_id*):
> {code}
> > select * from bulks_recipients where bulk_id = 
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;                                       
> >           
> ...
> (11 rows)
> {code}
> Ok, everything is fine.
> Now i'm doing *nodetool repair --partitioner-range --job-threads 4 --full* on 
> each node in cluster sequentially.
> After it finished:
> {code}
> > select * from bulks_recipients where bulk_id = 
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> ...
> (2 rows)
> {code}
> Only two rows.
> While the rows are actually there:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> If i issue an incremental repair on a random node, i can get like 7 rows 
> after index query.
> Dropping index and recreating it fixes the issue. Is it a bug or am i doing 
> the repair the wrong way?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to