Hello Joe,
These logs indicate the clocks are out of sync (by over 4.2 hours)
between the new node and the seed nodes:
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
Can you please check that the NTP client is running on all servers and
the clocks are in sync?
Cheers,
Bowen
On 10/09/2021 16:18, Joe Obernberger wrote:
Good idea.
There are two seed nodes:
I see this on one (note 172.16.100.44 is the new node):
DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
InboundConnectionInitiator.java:464 -
/172.16.100.44:7000(/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
messaging connection established, version = 12, framing = LZ4,
encryption = unencrypted
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65
- Pool Name Active Pending Completed
Blocked All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
- ReadStage 0 0 4729810 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
- CompactionExecutor 0 0 384171
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
- MutationStage 0 0 14540487 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- MemtableReclaimMemory 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- GossipStage 0 0 1126031 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- Native-Transport-Requests 0 0 13286230
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- RequestResponseStage 0 0 15724485
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- MemtableFlushWriter 0 0 298
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
- MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
- Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
- ValidationExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
- ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
- CacheCleanupExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79
- CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91
- MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:101
- Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:103
- KeyCache 75539240 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:109
- RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:116
- Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
- system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
- system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
- system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
- system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
- system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
- system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
- system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
- system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.node_operations 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.repair_run 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.repair_run_by_unit 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.diagnostic_event_subscription 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.running_reapers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.node_metrics_v3 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.repair_schedule_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
- reaper_db.repair_unit_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.repair_run_by_cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.schema_migration_leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.running_repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.schema_migration 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.node_metrics_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.repair_run_by_cluster_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- reaper_db.snapshot 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
- system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- system_traces.events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.doc 71463,9507451
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.fieldcounts 719515,2487290
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.meta_ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.parent_child 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.blocks 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.ds_30 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.child_parent 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.seq 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.doctemp 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
- doc.ds_21 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.extractedmetadata 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.datasources 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ds_16 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ordered_fieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ds_22 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.correlationbuckets 39374,8377095
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.binary_docs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ordered_metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.ds_27 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
- doc.correlationdocbuckets 450045,39051043
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.ds_29 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.ds_32 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.ds_31 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.correlation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.ds_20 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.indexorganize 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.binary_data 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.origdoc 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.test_ds16_ds17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
- doc.metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.ds_28 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.sourcebuckets 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.ds_18 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.sourcecorrelationcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.ds_17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
- doc.sourcecorrelation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65
- Pool Name Active Pending Completed
Blocked All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- ReadStage 0 0 4729901 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- CompactionExecutor 0 0 384171
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- MutationStage 0 0 14540850 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- MemtableReclaimMemory 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
- GossipStage 0 0 1126051 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- Native-Transport-Requests 0 0 13286521
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- RequestResponseStage 0 0 15724799
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- MemtableFlushWriter 0 0 298
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
- ValidationExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
- ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
- CacheCleanupExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79
- CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91
- MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:101
- Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:103
- KeyCache 75540688 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:109
- RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:116
- Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
- system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
- system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
- system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
- system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
- system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
- system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
- system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- reaper_db.node_operations 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
- reaper_db.repair_run 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.repair_run_by_unit 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.diagnostic_event_subscription 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.running_reapers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.node_metrics_v3 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.repair_schedule_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.repair_unit_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.repair_run_by_cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
- reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.schema_migration_leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.running_repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.schema_migration 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.node_metrics_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.repair_run_by_cluster_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- reaper_db.snapshot 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
- system_traces.events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.doc 71523,9515801
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.fieldcounts 719639,2487714
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.meta_ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.parent_child 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.blocks 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.ds_30 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.child_parent 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.seq 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.doctemp 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
- doc.ds_21 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.extractedmetadata 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.datasources 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.ds_16 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.ordered_fieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.ds_22 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.correlationbuckets 39427,8388316
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.binary_docs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
- doc.ordered_metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.ds_27 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.correlationdocbuckets 450171,39061883
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.ds_29 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.ds_32 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.ds_31 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.correlation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.ds_20 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.indexorganize 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.binary_data 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
- doc.origdoc 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.test_ds16_ds17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.ds_28 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.sourcebuckets 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.ds_18 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.sourcecorrelationcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.ds_17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
- doc.sourcecorrelation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65
- Pool Name Active Pending Completed
Blocked All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- ReadStage 0 0 4730049 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- CompactionExecutor 0 0 384171
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- MutationStage 0 0 14541404 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- MemtableReclaimMemory 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- GossipStage 0 0 1126070 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
- SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- Native-Transport-Requests 0 0 13286967
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- RequestResponseStage 0 0 15725284
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- MemtableFlushWriter 0 0 298
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- ValidationExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
- CacheCleanupExecutor 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79
- CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91
- MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:101
- Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:103
- KeyCache 75542896 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:109
- RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:116
- Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
- system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
- system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
- system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
- system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
- system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
- system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
- reaper_db.node_operations 0,0
.
.
.
and this on the other:
DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
InitialConnectionHandler.java:153 - Configured pipeline:
DefaultChannelPipeline{(frameDecoder =
org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
ExceptionHandlers.java:104 - Unknown exception in client networking
io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
failed: Connection reset by peer
DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
InitialConnectionHandler.java:121 - Response to STARTUP sent,
configuring pipeline for 5/v5
DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
InitialConnectionHandler.java:153 - Configured pipeline:
DefaultChannelPipeline{(frameDecoder =
org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
ExceptionHandlers.java:104 - Unknown exception in client networking
io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
failed: Connection reset by peer
INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
InboundConnectionInitiator.java:464 -
/172.16.100.44:7000(/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
messaging connection established, version = 12, framing = LZ4,
encryption = unencrypted
INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
.
.
.
-Joe
On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
Is there anything in the logs of the other nodes, particularly those
in the seeds list of the one which won't start? If this is a problem
with the peers being unable to respond, as the issue fixed in 4.0.1
(CASSANDRA-16877) was, you may see some indication there.
Sam
On 10 Sep 2021, at 15:56, Joe Obernberger
<joseph.obernber...@gmail.com> wrote:
Thank you Jeff - yes, this is on the latest 4.0.1
nodetool version
ReleaseVersion: 4.0.1
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.100.251 488.38 GiB 200 38.7%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN 172.16.100.208 76.02 GiB 30 5.8%
2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
UN 172.16.100.252 501.88 GiB 200 38.6%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN 172.16.100.249 517.27 GiB 200 38.6%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN 172.16.100.36 524.45 GiB 200 38.6%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN 172.16.100.39 521.05 GiB 200 38.6%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN 172.16.100.253 11.39 GiB 4 0.8%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN 172.16.100.248 524.46 GiB 200 38.7%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN 172.16.100.37 314.67 GiB 120 23.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN 172.16.100.250 464.23 GiB 200 38.6%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
yum list installed | grep cass
cassandra.noarch 4.0.1-1 @cassandra
-Joe
On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is
too large for the urgent message queue, causing this stack trace,
and was released 3 days ago. I've never seen it on a 10 node
cluster before, but I'd be trying that.
On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
<joseph.obernber...@gmail.com> wrote:
I have a 10 node cluster and am trying to add another node. The
new
node is running Rocky Linux and I'm getting the unable to
gossip with
any peers error. Firewall and SELinux are off. I can ping all the
other nodes OK. I've checked everything I can think of
(/etc/hosts,
listen_address, broadcast etc..). It all looks correct to me.
Any ideas? Could it be an incompatibility with Rocky?
DEBUG [main] 2021-09-10 06:45:24,846
YamlConfigurationLoader.java:112 -
Loading settings from
file:/etc/cassandra/default.conf/cassandra.yaml
INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
<http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
<http://172.16.100.37/>:7000-URGENT_MESSAGES-eae47864
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
TokenMetadata.java:525
- Updating topology for all endpoints that have changed
DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
recording size estimates
ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
at
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
StorageService.java:1621 - DRAINING: starting drain process
INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
Gossiper.java:1993 - No local state, state is in silent
shutdown, or
node hasn't joined, not announcing shutdown
Thank you!
-Joe
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Virus-free. www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<x-msg://5/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
