Re: Is my range read query behaving strange ?

[Laxmikant Upadhyay]

Raised a ticket  https://issues.apache.org/jira/browse/CASSANDRA-15159 for
the same.

On Thu, Jun 13, 2019 at 3:55 PM Laxmikant Upadhyay 
wrote:

> This issue is reproducible on *3.11.4 and 2.1.21* as well.  (not yet
> checked on  3.0)
>
> Range query could be : select * from test.table1;  (In this case, Read
> repair actually sending old mutation to the node which has tombstone  )
> I also ran normal read and it also returns a row in this case instead of
> empty response.
>
> cqlsh> select * from test.table1;
>
>  key  | col | val
> --+-+-
>  key2 | abc | xyz
>
> (1 rows)
> cqlsh> select * from test.table1 where key = 'key2';
>
>  key  | col | val
> --+-+-
>  key2 | abc | xyz
>
> Why is this behaviour ?  Why read is ignoring purge-able tombstone ?
>
>
>
> On Thu, Jun 13, 2019 at 12:30 PM Laxmikant Upadhyay <
> laxmikant@gmail.com> wrote:
>
>> HI Michael,
>>
>> Thanks for your reply.
>> I don't think this issue is related to CASSANDRA-12765
>>  as in my case
>> the sstable which has tombstone does not have maxLocalDeletionTime ==
>> Integer.MAX_VALUE .  I am able to reproduce this issue on 2.1.17 as well.
>>
>> I am attaching the steps to reproduce on 2.1.17 (with minor change from
>> previous steps to make sure one request must go to the node which has old
>> mutation). I have also attached the trace of range read query.
>>
>> Should I raise a jira for the same ?
>>
>> On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler 
>> wrote:
>>
>>> (dropped dev@ x-post; user@ was correct)
>>>
>>> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest
>>> 2.1.21.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-12765
>>> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>>>
>>> Michael
>>>
>>> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
>>> > Does range query ignore purgable tombstone (which crossed grace
>>> period)
>>> > in some cases?
>>> >
>>> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
>>> > mailto:laxmikant@gmail.com>> wrote:
>>> >
>>> > In a 3 node cassandra 2.1.16 cluster where, one node has old
>>> > mutation and two nodes have evict-able (crossed gc grace period)
>>> > tombstone produced by TTL. A read range  query with local quorum
>>> > return the old mutation as result. However expected result should
>>> be
>>> > empty. Next time running the same query results no data as
>>> expected.
>>> > Why this strange behaviour?
>>> >
>>> >
>>> > *Steps to Reproduce :*
>>> > Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff
>>> > for each node.
>>> >
>>> > #ccm node1 nodetool ring
>>> > Datacenter: datacenter1
>>> > ==
>>> > AddressRackStatus State   LoadOwns
>>>
>>> >   Token
>>> >
>>>
>>> >3074457345618258602
>>> > 127.0.0.1  rack1   Up Normal  175.12 KB   100.00%
>>> >-9223372036854775808
>>> > 127.0.0.2  rack1   Up   Normal  177.87 KB   100.00%
>>> >  -3074457345618258603
>>> > 127.0.0.3  rack1   Up   Normal  175.13 KB   100.00%
>>> >  3074457345618258602
>>> >
>>> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>>> >
>>> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
>>> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
>>> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
>>> > text,PRIMARY KEY ((key), col));
>>> > cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;
>>> >
>>> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
>>> > 'abc','xyz');
>>> >
>>> > #ccm flush
>>> >
>>> > #ccm node3 stop
>>> >
>>> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
>>> > 'abc','xyz') USING TTL 60;
>>> >
>>> > #ccm flush
>>> >
>>> > #wait for 3 min so that the tombstone crosses its gc grace period.
>>> >
>>> > #ccm node3 start
>>> >
>>> > cqlsh> select * from test.table1 where token (key) >
>>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>>> >
>>> >   key  | col | val
>>> > --+-+-
>>> >   key2 | abc | xyz
>>> >
>>> > (1 rows)
>>> >
>>> > #ccm flush
>>> > -> Here read repair triggers and the old mutation moves to the one
>>> > of the node where tombstone is present (not both the node)
>>> >
>>> >
>>> > cqlsh> select * from test.vouchers where token (key) >
>>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>>> >
>>> >   key | col | val
>>> > -+-+-
>>> >
>>> > (0 rows)
>>> >
>>> >
>>> > --
>>> >
>>> > regards,
>>> > Laxmikant Upadhyay
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandr

Re: Is my range read query behaving strange ?

[Laxmikant Upadhyay]

This issue is reproducible on *3.11.4 and 2.1.21* as well.  (not yet
checked on  3.0)

Range query could be : select * from test.table1;  (In this case, Read
repair actually sending old mutation to the node which has tombstone  )
I also ran normal read and it also returns a row in this case instead of
empty response.

cqlsh> select * from test.table1;

 key  | col | val
--+-+-
 key2 | abc | xyz

(1 rows)
cqlsh> select * from test.table1 where key = 'key2';

 key  | col | val
--+-+-
 key2 | abc | xyz

Why is this behaviour ?  Why read is ignoring purge-able tombstone ?



On Thu, Jun 13, 2019 at 12:30 PM Laxmikant Upadhyay 
wrote:

> HI Michael,
>
> Thanks for your reply.
> I don't think this issue is related to CASSANDRA-12765
>  as in my case the
> sstable which has tombstone does not have maxLocalDeletionTime ==
> Integer.MAX_VALUE .  I am able to reproduce this issue on 2.1.17 as well.
>
> I am attaching the steps to reproduce on 2.1.17 (with minor change from
> previous steps to make sure one request must go to the node which has old
> mutation). I have also attached the trace of range read query.
>
> Should I raise a jira for the same ?
>
> On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler 
> wrote:
>
>> (dropped dev@ x-post; user@ was correct)
>>
>> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest
>> 2.1.21.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-12765
>> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>>
>> Michael
>>
>> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
>> > Does range query ignore purgable tombstone (which crossed grace period)
>> > in some cases?
>> >
>> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
>> > mailto:laxmikant@gmail.com>> wrote:
>> >
>> > In a 3 node cassandra 2.1.16 cluster where, one node has old
>> > mutation and two nodes have evict-able (crossed gc grace period)
>> > tombstone produced by TTL. A read range  query with local quorum
>> > return the old mutation as result. However expected result should be
>> > empty. Next time running the same query results no data as expected.
>> > Why this strange behaviour?
>> >
>> >
>> > *Steps to Reproduce :*
>> > Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff
>> > for each node.
>> >
>> > #ccm node1 nodetool ring
>> > Datacenter: datacenter1
>> > ==
>> > AddressRackStatus State   LoadOwns
>> >   Token
>> >
>> >3074457345618258602
>> > 127.0.0.1  rack1   Up Normal  175.12 KB   100.00%
>> >-9223372036854775808
>> > 127.0.0.2  rack1   Up   Normal  177.87 KB   100.00%
>> >  -3074457345618258603
>> > 127.0.0.3  rack1   Up   Normal  175.13 KB   100.00%
>> >  3074457345618258602
>> >
>> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>> >
>> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
>> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
>> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
>> > text,PRIMARY KEY ((key), col));
>> > cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;
>> >
>> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
>> > 'abc','xyz');
>> >
>> > #ccm flush
>> >
>> > #ccm node3 stop
>> >
>> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
>> > 'abc','xyz') USING TTL 60;
>> >
>> > #ccm flush
>> >
>> > #wait for 3 min so that the tombstone crosses its gc grace period.
>> >
>> > #ccm node3 start
>> >
>> > cqlsh> select * from test.table1 where token (key) >
>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>> >
>> >   key  | col | val
>> > --+-+-
>> >   key2 | abc | xyz
>> >
>> > (1 rows)
>> >
>> > #ccm flush
>> > -> Here read repair triggers and the old mutation moves to the one
>> > of the node where tombstone is present (not both the node)
>> >
>> >
>> > cqlsh> select * from test.vouchers where token (key) >
>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>> >
>> >   key | col | val
>> > -+-+-
>> >
>> > (0 rows)
>> >
>> >
>> > --
>> >
>> > regards,
>> > Laxmikant Upadhyay
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>

-- 

regards,
Laxmikant Upadhyay

Re: Is my range read query behaving strange ?

[Laxmikant Upadhyay]

HI Michael,

Thanks for your reply.
I don't think this issue is related to CASSANDRA-12765
 as in my case the
sstable which has tombstone does not have maxLocalDeletionTime ==
Integer.MAX_VALUE .  I am able to reproduce this issue on 2.1.17 as well.

I am attaching the steps to reproduce on 2.1.17 (with minor change from
previous steps to make sure one request must go to the node which has old
mutation). I have also attached the trace of range read query.

Should I raise a jira for the same ?

On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler 
wrote:

> (dropped dev@ x-post; user@ was correct)
>
> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest 2.1.21.
>
> https://issues.apache.org/jira/browse/CASSANDRA-12765
> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>
> Michael
>
> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
> > Does range query ignore purgable tombstone (which crossed grace period)
> > in some cases?
> >
> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
> > mailto:laxmikant@gmail.com>> wrote:
> >
> > In a 3 node cassandra 2.1.16 cluster where, one node has old
> > mutation and two nodes have evict-able (crossed gc grace period)
> > tombstone produced by TTL. A read range  query with local quorum
> > return the old mutation as result. However expected result should be
> > empty. Next time running the same query results no data as expected.
> > Why this strange behaviour?
> >
> >
> > *Steps to Reproduce :*
> > Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff
> > for each node.
> >
> > #ccm node1 nodetool ring
> > Datacenter: datacenter1
> > ==
> > AddressRackStatus State   LoadOwns
> >   Token
> >
> >3074457345618258602
> > 127.0.0.1  rack1   Up Normal  175.12 KB   100.00%
> >-9223372036854775808
> > 127.0.0.2  rack1   Up   Normal  177.87 KB   100.00%
> >  -3074457345618258603
> > 127.0.0.3  rack1   Up   Normal  175.13 KB   100.00%
> >  3074457345618258602
> >
> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
> >
> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
> > text,PRIMARY KEY ((key), col));
> > cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;
> >
> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
> > 'abc','xyz');
> >
> > #ccm flush
> >
> > #ccm node3 stop
> >
> > cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
> > 'abc','xyz') USING TTL 60;
> >
> > #ccm flush
> >
> > #wait for 3 min so that the tombstone crosses its gc grace period.
> >
> > #ccm node3 start
> >
> > cqlsh> select * from test.table1 where token (key) >
> > 3074457345618258602 and token (key) < -9223372036854775808 ;
> >
> >   key  | col | val
> > --+-+-
> >   key2 | abc | xyz
> >
> > (1 rows)
> >
> > #ccm flush
> > -> Here read repair triggers and the old mutation moves to the one
> > of the node where tombstone is present (not both the node)
> >
> >
> > cqlsh> select * from test.vouchers where token (key) >
> > 3074457345618258602 and token (key) < -9223372036854775808 ;
> >
> >   key | col | val
> > -+-+-
> >
> > (0 rows)
> >
> >
> > --
> >
> > regards,
> > Laxmikant Upadhyay
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

-- 

regards,
Laxmikant Upadhyay
# Create a ccm cluster on version 2.1.17

  # ccm create test1 -v 2.1.17
  
# Populate cluster with 3 nodes.

  # ccm populate -n 3
  
# Disable hinted hand off in all nodes of the cluster 
# Start all the nodes of the cluster

  #ccm start

# Run the nodetool ring on any of the node to find the token ranges.

  # ccm node1 nodetool ring
  
Datacenter: datacenter1
==
AddressRackStatus State   LoadOwnsToken
  
3074457345618258602
127.0.0.1  rack1   Up Normal  46.6 KB 66.67%  
-9223372036854775808
127.0.0.2  rack1   Up Normal  91.86 KB66.67%  
-3074457345618258603
127.0.0.3  rack1   Up Normal  46.65 KB66.67%  
3074457345618258602

# Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;

  # ccm node1 cqlsh
  cqlsh> CONISISTENCY LOCAL_QUORUM;
  
# Create keyspace with replication factor 3
   
   cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' : 
'NetworkTopologySt

Re: Is my range read query behaving strange ?

[Michael Shuler]

(dropped dev@ x-post; user@ was correct)

Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest 2.1.21.

https://issues.apache.org/jira/browse/CASSANDRA-12765
https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36

Michael

On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
Does range query ignore purgable tombstone (which crossed grace period) 
in some cases?


On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay 
mailto:laxmikant@gmail.com>> wrote:


In a 3 node cassandra 2.1.16 cluster where, one node has old
mutation and two nodes have evict-able (crossed gc grace period)
tombstone produced by TTL. A read range  query with local quorum
return the old mutation as result. However expected result should be
empty. Next time running the same query results no data as expected.
Why this strange behaviour?


*Steps to Reproduce :*
Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff
for each node.

#ccm node1 nodetool ring
Datacenter: datacenter1
==
Address    Rack        Status State   Load            Owns  
      Token

       3074457345618258602
127.0.0.1  rack1       Up     Normal  175.12 KB       100.00%  
       -9223372036854775808
127.0.0.2  rack1       Up   Normal  177.87 KB       100.00%
     -3074457345618258603
127.0.0.3  rack1       Up   Normal  175.13 KB       100.00%
     3074457345618258602


#Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;

cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
cqlsh> CREATE TABLE test.table1 (key text, col text, val
text,PRIMARY KEY ((key), col));
cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;

cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
'abc','xyz');

#ccm flush

#ccm node3 stop

cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
'abc','xyz') USING TTL 60;

#ccm flush

#wait for 3 min so that the tombstone crosses its gc grace period.

#ccm node3 start

cqlsh> select * from test.table1 where token (key) >
3074457345618258602 and token (key) < -9223372036854775808 ;

  key  | col | val
--+-+-
  key2 | abc | xyz

(1 rows)

#ccm flush
-> Here read repair triggers and the old mutation moves to the one
of the node where tombstone is present (not both the node)


cqlsh> select * from test.vouchers where token (key) >
3074457345618258602 and token (key) < -9223372036854775808 ;

  key | col | val
-+-+-

(0 rows)


-- 


regards,
Laxmikant Upadhyay




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Is my range read query behaving strange ?

[Laxmikant Upadhyay]

Does range query ignore purgable tombstone (which crossed grace period) in
some cases?

On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay 
wrote:

> In a 3 node cassandra 2.1.16 cluster where, one node has old mutation and
> two nodes have evict-able (crossed gc grace period) tombstone produced by
> TTL. A read range  query with local quorum return the old mutation as
> result. However expected result should be empty. Next time running the same
> query results no data as expected. Why this strange behaviour?
>
>
> *Steps to Reproduce :*
> Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff for each
> node.
>
> #ccm node1 nodetool ring
> Datacenter: datacenter1
> ==
> AddressRackStatus State   LoadOwns
>  Token
>
> 3074457345618258602
> 127.0.0.1  rack1   Up Normal  175.12 KB   100.00%
> -9223372036854775808
> 127.0.0.2  rack1   Up   Normal  177.87 KB   100.00%
> -3074457345618258603
> 127.0.0.3  rack1   Up   Normal  175.13 KB   100.00%
> 3074457345618258602
>
> #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>
> cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' :
> 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> cqlsh> CREATE TABLE test.table1 (key text, col text, val text,PRIMARY KEY
> ((key), col));
> cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;
>
> cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
> 'abc','xyz');
>
> #ccm flush
>
> #ccm node3 stop
>
> cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
> 'abc','xyz') USING TTL 60;
>
> #ccm flush
>
> #wait for 3 min so that the tombstone crosses its gc grace period.
>
> #ccm node3 start
>
> cqlsh> select * from test.table1 where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
>  key  | col | val
> --+-+-
>  key2 | abc | xyz
>
> (1 rows)
>
> #ccm flush
> -> Here read repair triggers and the old mutation moves to the one of the
> node where tombstone is present (not both the node)
>
>
> cqlsh> select * from test.vouchers where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
>  key | col | val
> -+-+-
>
> (0 rows)
>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>

Is my range read query behaving strange ?

[Laxmikant Upadhyay]

In a 3 node cassandra 2.1.16 cluster where, one node has old mutation and
two nodes have evict-able (crossed gc grace period) tombstone produced by
TTL. A read range  query with local quorum return the old mutation as
result. However expected result should be empty. Next time running the same
query results no data as expected. Why this strange behaviour?


*Steps to Reproduce :*
Create a cassandra-2.1.16  3 node cluster. Disable hinted handoff for each
node.

#ccm node1 nodetool ring
Datacenter: datacenter1
==
AddressRackStatus State   LoadOwns
 Token

3074457345618258602
127.0.0.1  rack1   Up Normal  175.12 KB   100.00%
-9223372036854775808
127.0.0.2  rack1   Up   Normal  177.87 KB   100.00%
-3074457345618258603
127.0.0.3  rack1   Up   Normal  175.13 KB   100.00%
3074457345618258602

#Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;

cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'datacenter1' : 3 };
cqlsh> CREATE TABLE test.table1 (key text, col text, val text,PRIMARY KEY
((key), col));
cqlsh> ALTER TABLE test.table1  with GC_GRACE_SECONDS = 120;

cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
'abc','xyz');

#ccm flush

#ccm node3 stop

cqlsh> INSERT INTO test.table1  (key, col, val) VALUES ('key2',
'abc','xyz') USING TTL 60;

#ccm flush

#wait for 3 min so that the tombstone crosses its gc grace period.

#ccm node3 start

cqlsh> select * from test.table1 where token (key) > 3074457345618258602
and token (key) < -9223372036854775808 ;

 key  | col | val
--+-+-
 key2 | abc | xyz

(1 rows)

#ccm flush
-> Here read repair triggers and the old mutation moves to the one of the
node where tombstone is present (not both the node)


cqlsh> select * from test.vouchers where token (key) > 3074457345618258602
and token (key) < -9223372036854775808 ;

 key | col | val
-+-+-

(0 rows)


-- 

regards,
Laxmikant Upadhyay