[
https://issues.apache.org/jira/browse/CASSANDRA-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yifan Cai updated CASSANDRA-16345:
----------------------------------
Test and Documentation Plan:
Test Plan
# Create 4 nodes cluster and create keyspace with RF == 3
# Run data prepopulation
# Run steady state load for X seconds as Phase 1.
# Trigger token move and wait for completion
# Let the traffic continue for Y seconds, then trigger relocatesstable and
wait for completion
# Let the traffic continue for Z seconds, then trigger relocatesstable again.
The expectation is that the second relocatesstable should be a no-op.
The workload used in the test was generated from tlp-stress.
Steady state load Read : Write : Delete == 5 : 4 : 1, and the QPS was kept at
3K/s.
Status: Patch Available (was: Open)
Posting the filtered logs from the test.
For all 4 test cases, the log shows the second relocatesstable does not
relocate. It mean that all sstables are in the correct location. (The IPs of
the peers were redacted)
*STCS with 5 data directories*
{code:java}
➜ grep -E 'MOVING|\[Stream #|Relocate|RELOCATE'
cassandra-5d-relocatesstables-stcs.hlog2021-01-06 01:02:37,630 INFO [RMI TCP
Connection(80)-127.0.0.1] org.apache.cassandra.service.StorageService - MOVING:
Moving pod-111000301-7a757362-e7a1-4f57-a839-38fd513f49ad/IP-1:7000 from
1148914691236517205 to 2148914691236517205.2021-01-06 01:02:37,630 INFO [RMI
TCP Connection(80)-127.0.0.1] org.apache.cassandra.service.StorageService -
MOVING: Sleeping 30000 ms before start streaming/fetching ranges2021-01-06
01:03:07,633 INFO [RMI TCP Connection(80)-127.0.0.1]
org.apache.cassandra.service.StorageService - MOVING: fetching new ranges and
streaming old ranges2021-01-06 01:03:07,639 INFO [RMI TCP
Connection(80)-127.0.0.1] org.apache.cassandra.streaming.StreamResultFuture -
[Stream #f02378e0-4fba-11eb-bc08-d151c09ee185] Executing streaming plan for
Relocation2021-01-06 01:03:07,640 INFO [RMI TCP Connection(80)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185] Starting streaming to
/IP-2:70002021-01-06 01:03:07,654 INFO [RMI TCP Connection(80)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185, ID#0] Beginning stream session with
/IP-2:70002021-01-06 01:03:07,654 INFO [RMI TCP Connection(80)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185] Starting streaming to
/IP-3:70002021-01-06 01:03:07,665 INFO [RMI TCP Connection(80)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185, ID#0] Beginning stream session with
/IP-3:70002021-01-06 01:03:08,855 INFO
[Stream-Deserializer-/IP-2:7000-65abd603]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185 ID#0] Prepare completed. Receiving 20
files(11.212GiB), sending 0 files(0.000KiB)2021-01-06 01:03:08,866 INFO
[Stream-Deserializer-/IP-3:7000-1507f409]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185 ID#0] Prepare completed. Receiving 0
files(0.000KiB), sending 0 files(0.000KiB)2021-01-06 01:03:09,267 INFO
[Stream-Deserializer-/IP-3:7000-1507f409]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185] Session with /IP-3:7000 is
complete2021-01-06 01:10:49,946 INFO [Stream-Deserializer-/IP-2:7000-65abd603]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185] Session with /IP-2:7000 is
complete2021-01-06 01:10:49,946 INFO [Stream-Deserializer-/IP-2:7000-65abd603]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#f02378e0-4fba-11eb-bc08-d151c09ee185] All sessions completed2021-01-06
01:11:03,747 INFO [RMI TCP Connection(91)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-06
01:29:57,476 INFO [RMI TCP Connection(91)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_2
successfully2021-01-06 01:40:42,123 INFO [RMI TCP Connection(111)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-06
01:40:42,124 INFO [RMI TCP Connection(111)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE
for tlp_stress.keyvaluelargeblob_2
{code}
*STCS with 20 data directories*
{code:java}
➜ grep -E 'MOVING|\[Stream #|Relocate|RELOCATE'
cassandra-20d-relocatesstables-stcs.hlog2021-01-08 07:30:56,554 INFO [RMI TCP
Connection(216)-127.0.0.1] org.apache.cassandra.service.StorageService -
MOVING: Moving pod-120000401-d9b7a3ed-4e3a-4ad4-9cdc-7a188abda134/IP-1:7000
from 4611686018427387904 to 5611686018427387904.2021-01-08 07:30:56,555 INFO
[RMI TCP Connection(216)-127.0.0.1] org.apache.cassandra.service.StorageService
- MOVING: Sleeping 30000 ms before start streaming/fetching ranges2021-01-08
07:31:26,568 INFO [RMI TCP Connection(216)-127.0.0.1]
org.apache.cassandra.service.StorageService - MOVING: fetching new ranges and
streaming old ranges2021-01-08 07:31:26,588 INFO [RMI TCP
Connection(216)-127.0.0.1] org.apache.cassandra.streaming.StreamResultFuture -
[Stream #843518d0-5183-11eb-b6d9-171a206caaff] Executing streaming plan for
Relocation2021-01-08 07:31:26,589 INFO [RMI TCP Connection(216)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff] Starting streaming to
/IP-2:70002021-01-08 07:31:26,603 INFO [RMI TCP Connection(216)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff, ID#0] Beginning stream session with
/IP-2:70002021-01-08 07:31:26,603 INFO [RMI TCP Connection(216)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff] Starting streaming to
/IP-3:70002021-01-08 07:31:26,613 INFO [RMI TCP Connection(216)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff, ID#0] Beginning stream session with
/IP-3:70002021-01-08 07:31:27,804 INFO
[Stream-Deserializer-/IP-2:7000-b0b1f594]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff ID#0] Prepare completed. Receiving 0
files(0.000KiB), sending 0 files(0.000KiB)2021-01-08 07:31:27,804 INFO
[Stream-Deserializer-/IP-2:7000-b0b1f594]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff] Session with /IP-2:7000 is
complete2021-01-08 07:31:28,214 INFO [Stream-Deserializer-/IP-3:7000-4a1c2822]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff ID#0] Prepare completed. Receiving 55
files(11.111GiB), sending 0 files(0.000KiB)2021-01-08 07:39:11,934 INFO
[Stream-Deserializer-/IP-3:7000-4a1c2822]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff] Session with /IP-3:7000 is
complete2021-01-08 07:39:11,934 INFO [Stream-Deserializer-/IP-3:7000-4a1c2822]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#843518d0-5183-11eb-b6d9-171a206caaff] All sessions completed2021-01-08
07:54:17,557 INFO [RMI TCP Connection(243)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-08
08:21:26,307 INFO [RMI TCP Connection(243)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_2
successfully2021-01-08 08:51:31,537 INFO [RMI TCP Connection(315)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-08
08:51:31,545 INFO [RMI TCP Connection(315)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE
for tlp_stress.keyvaluelargeblob_2
{code}
LCS with 5 data directories
{code:java}
➜ grep -E 'MOVING|\[Stream #|Relocate|RELOCATE'
cassandra-5d-relocatesstables-lcs.hlog2021-01-07 06:28:29,661 INFO [RMI TCP
Connection(253)-127.0.0.1] org.apache.cassandra.service.StorageService -
MOVING: Moving pod-105000401-f569c317-44ec-4bee-b64c-a44b9c181157/IP-1:7000
from 4611686018427387904 to 5611686018427387904.2021-01-07 06:28:29,662 INFO
[RMI TCP Connection(253)-127.0.0.1] org.apache.cassandra.service.StorageService
- MOVING: Sleeping 30000 ms before start streaming/fetching ranges2021-01-07
06:28:59,676 INFO [RMI TCP Connection(253)-127.0.0.1]
org.apache.cassandra.service.StorageService - MOVING: fetching new ranges and
streaming old ranges2021-01-07 06:28:59,700 INFO [RMI TCP
Connection(253)-127.0.0.1] org.apache.cassandra.streaming.StreamResultFuture -
[Stream #a078d900-50b1-11eb-85e2-7f98aaf9f061] Executing streaming plan for
Relocation2021-01-07 06:28:59,700 INFO [RMI TCP Connection(253)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061] Starting streaming to
/IP-2:70002021-01-07 06:28:59,714 INFO [RMI TCP Connection(253)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061, ID#0] Beginning stream session with
/IP-2:70002021-01-07 06:28:59,714 INFO [RMI TCP Connection(253)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061] Starting streaming to
/IP-3:70002021-01-07 06:28:59,726 INFO [RMI TCP Connection(253)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061, ID#0] Beginning stream session with
/IP-3:70002021-01-07 06:29:00,928 INFO
[Stream-Deserializer-/IP-3:7000-1b712ebe]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061 ID#0] Prepare completed. Receiving 736
files(11.240GiB), sending 0 files(0.000KiB)2021-01-07 06:29:01,315 INFO
[Stream-Deserializer-/IP-2:7000-e69d9d44]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061 ID#0] Prepare completed. Receiving 0
files(0.000KiB), sending 0 files(0.000KiB)2021-01-07 06:29:01,315 INFO
[Stream-Deserializer-/IP-2:7000-e69d9d44]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061] Session with /IP-2:7000 is
complete2021-01-07 06:36:42,740 INFO [Stream-Deserializer-/IP-3:7000-1b712ebe]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061] Session with /IP-3:7000 is
complete2021-01-07 06:36:42,740 INFO [Stream-Deserializer-/IP-3:7000-1b712ebe]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#a078d900-50b1-11eb-85e2-7f98aaf9f061] All sessions completed2021-01-07
06:43:41,361 INFO [RMI TCP Connection(273)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-07
06:47:56,486 INFO [RMI TCP Connection(273)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_2
successfully2021-01-07 07:18:01,476 INFO [RMI TCP Connection(298)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-07
07:18:01,844 INFO [RMI TCP Connection(298)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE
for tlp_stress.keyvaluelargeblob_2
{code}
*LCS with 20 data directories*
{code:java}
➜ grep -E 'MOVING|\[Stream #|Relocate|RELOCATE'
cassandra-20d-relocatesstables-lcs.hlog2021-01-08 07:31:09,839 INFO [RMI TCP
Connection(62)-127.0.0.1] org.apache.cassandra.service.StorageService - MOVING:
Moving pod-120000401-bf853a7c-b260-44eb-99e8-b3d04f8d6a71/IP-1:7000 from
4611686018427387904 to 5611686018427387904.2021-01-08 07:31:09,840 INFO [RMI
TCP Connection(62)-127.0.0.1] org.apache.cassandra.service.StorageService -
MOVING: Sleeping 30000 ms before start streaming/fetching ranges2021-01-08
07:31:39,860 INFO [RMI TCP Connection(62)-127.0.0.1]
org.apache.cassandra.service.StorageService - MOVING: fetching new ranges and
streaming old ranges2021-01-08 07:31:39,888 INFO [RMI TCP
Connection(62)-127.0.0.1] org.apache.cassandra.streaming.StreamResultFuture -
[Stream #8c206130-5183-11eb-a261-87d9e37f4503] Executing streaming plan for
Relocation2021-01-08 07:31:39,888 INFO [RMI TCP Connection(62)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503] Starting streaming to
/IP-2:70002021-01-08 07:31:39,910 INFO [RMI TCP Connection(62)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503, ID#0] Beginning stream session with
/IP-2:70002021-01-08 07:31:39,910 INFO [RMI TCP Connection(62)-127.0.0.1]
org.apache.cassandra.streaming.StreamSession - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503] Starting streaming to
/IP-3:70002021-01-08 07:31:39,928 INFO [RMI TCP Connection(62)-127.0.0.1]
org.apache.cassandra.streaming.StreamCoordinator - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503, ID#0] Beginning stream session with
/IP-3:70002021-01-08 07:31:40,728 INFO
[Stream-Deserializer-/IP-3:7000-f1acb3b8]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503 ID#0] Prepare completed. Receiving 0
files(0.000KiB), sending 0 files(0.000KiB)2021-01-08 07:31:40,729 INFO
[Stream-Deserializer-/IP-3:7000-f1acb3b8]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503] Session with /IP-3:7000 is
complete2021-01-08 07:31:41,916 INFO [Stream-Deserializer-/IP-2:7000-c8066789]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503 ID#0] Prepare completed. Receiving 611
files(11.157GiB), sending 0 files(0.000KiB)2021-01-08 07:39:19,896 INFO
[Stream-Deserializer-/IP-2:7000-c8066789]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503] Session with /IP-2:7000 is
complete2021-01-08 07:39:19,896 INFO [Stream-Deserializer-/IP-2:7000-c8066789]
org.apache.cassandra.streaming.StreamResultFuture - [Stream
#8c206130-5183-11eb-a261-87d9e37f4503] All sessions completed2021-01-08
07:54:25,929 INFO [RMI TCP Connection(76)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-08
08:12:48,995 INFO [RMI TCP Connection(76)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_2
successfully2021-01-08 08:42:54,232 INFO [RMI TCP Connection(107)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate
sstables to correct disk for tlp_stress.keyvaluelargeblob_22021-01-08
08:42:54,600 INFO [RMI TCP Connection(107)-127.0.0.1]
org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE
for tlp_stress.keyvaluelargeblob_2
{code}
> Test SSTables are in the correct location after range movement with 1/5/20
> data directories
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16345
> Project: Cassandra
> Issue Type: Sub-task
> Components: Test/benchmark
> Reporter: Yifan Cai
> Assignee: Yifan Cai
> Priority: Normal
>
> The testing cluster should be pre-populated with ~200GB data in each node.
> Change the token ranges and verify that each data directory only contains
> SSTables belongs to it. In other word, no SSTables are relocated by running
> nodetool "relocatesstables". Both LCS and STCS need to be covered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
