Since 3.x worked and 4.x didn't, it is most likely caused by clock out
of sync between servers. Cassandra 3.x by default has cross node timeout
disabled, and Cassandra 4 has changed the default to enabled. Check the
system time on both servers are accurate. Use NTP to prevent this from
happening again.
On 31/01/2022 10:30, onmstester onmstester wrote:
Hi,
I'm trying to setup a Cluster of apache Cassandra version 4.0.1 with
2 nodes:
1. on node1 (192.168.1.1), extracted tar.gz and config these on yml:
- seeds: "192.168.1.1"
listen_address: 192.168.1.1
rpc_address: 192.168.1.1
2. started node1 and a few seconds later it is UN
3.on node2 (192.168.1.2), extracted tar.gz and config these on yml:
- seeds: "192.168.1.1"
listen_address: 192.168.1.2
rpc_address: 192.168.1.2
4. started node2 but a few seconds later an error reported in system.log:
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,499
MessagingMetrics.java:206 - ECHO_REQ messages were dropped in last
5000 ms: 0 internal and 5 cross node. Mean internal dropped latency: 0
ms and Mean cross-node dropped latency: 53142 ms
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,499 StatusLogger.java:65
- Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,503 StatusLogger.java:69
- CompactionExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,503 StatusLogger.java:69
- MemtableReclaimMemory 0 0 4
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,503 StatusLogger.java:69
- GossipStage 0 0 24 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,503 StatusLogger.java:69
- SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,503 StatusLogger.java:69
- HintsDispatcher 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- MemtableFlushWriter 0 0 4
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- PerDiskMemtableFlushWriter_0 0 0 4
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- MemtablePostFlush 0 0 8 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- ValidationExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,504 StatusLogger.java:69
- ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,505 StatusLogger.java:69
- CacheCleanupExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,505 StatusLogger.java:79
- CompactionManager 0 0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,505 StatusLogger.java:91
- MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:101
- Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:103
- KeyCache 240 104857600 all
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:109
- RowCache 0 0 all
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:116
- Table Memtable ops,data
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:119
- system_schema.columns 218,33711
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,506 StatusLogger.java:119
- system_schema.types 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.indexes 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.keyspaces 4,242
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.dropped_columns 3,123
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.aggregates 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.triggers 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.tables 32,22085
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.views 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,507 StatusLogger.java:119
- system_schema.functions 2,16
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.compaction_history 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.repairs 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.size_estimates 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.table_estimates 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.paxos 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.built_views 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,508 StatusLogger.java:119
- system.peer_events 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.peers_v2 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.peers 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.batches 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,509 StatusLogger.java:119
- system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,510 StatusLogger.java:119
- system.local 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,510 StatusLogger.java:119
- system.sstable_activity 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,510 StatusLogger.java:119
- system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,510 StatusLogger.java:119
- system.available_ranges 0,0
INFO [ScheduledTasks:1] 2022-01-31 13:46:37,510 StatusLogger.java:119
- system.prepared_statements 0,0
Checked out some other scenarios:
- tried to setup a cluster with same procedure using Cassandra 3.11.2
and it started with no problems (so there is no problem related to
Hardware or OS)
- single node cluster setup on node2 starting with no problems
- tried to set both nodes as seed node but got 100s of these logs per
seconds and still no luck:
WARN [MigrationStage:1] 2022-01-31 13:38:23,872
MigrationCoordinator.java:504 - Can't send schema pull request: node
/192.168.1.1:7000 is down.
DEBUG [MigrationStage:1] 2022-01-31 13:38:23,872
MigrationCoordinator.java:324 - Immediately submitting migration task
for /192.168.1.1:7000, schema versions: local=(empty),
remote=2207c2a9-f598-3971-986b-2926e09e239d
Cassandra version: 4.0.1
OS: ubunutu-20 LTS
java: 8u321 oracle
Best Regards
Sent using Zoho Mail <https://www.zoho.com/mail/>
