Hmm. are the ports open on the `new` server?
Looks like it can connect to other nodes but other nodes can't connect to
it.
-Vy
On Fri, Sep 10, 2021 at 10:20 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Good idea.
> There are two seed nodes:
> I see this on one (note 172.16.100.44 is the new node):
>
> DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
> InboundConnectionInitiator.java:464 - /172.16.100.44:7000
> (/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
> messaging connection established, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> ReadStage 0 0 4729810
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> MutationStage 0 0 14540487
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> GossipStage 0 0 1126031
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286230
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> RequestResponseStage 0 0 15724485
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:103 -
> KeyCache 75539240
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.doc 71463,9507451
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.fieldcounts 719515,2487290
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.correlationbuckets 39374,8377095
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.correlationdocbuckets 450045,39051043
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> ReadStage 0 0 4729901
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> MutationStage 0 0 14540850
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> GossipStage 0 0 1126051
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286521
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> RequestResponseStage 0 0 15724799
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:103 -
> KeyCache 75540688
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.doc 71523,9515801
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.fieldcounts 719639,2487714
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.correlationbuckets 39427,8388316
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.correlationdocbuckets 450171,39061883
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> ReadStage 0 0 4730049
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> MutationStage 0 0 14541404
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> GossipStage 0 0 1126070
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286967
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> RequestResponseStage 0 0 15725284
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:103 -
> KeyCache 75542896
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> .
> .
> .
>
> and this on the other:
>
> DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
> Connection reset by peer
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
> InitialConnectionHandler.java:121 - Response to STARTUP sent, configuring
> pipeline for 5/v5
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
> Connection reset by peer
> INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
> InboundConnectionInitiator.java:464 - /172.16.100.44:7000
> (/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
> messaging connection established, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> .
> .
> .
>
> -Joe
>
>
> On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
>
> Is there anything in the logs of the other nodes, particularly those in
> the seeds list of the one which won't start? If this is a problem with the
> peers being unable to respond, as the issue fixed in 4.0.1
> (CASSANDRA-16877) was, you may see some indication there.
>
> Sam
>
> On 10 Sep 2021, at 15:56, Joe Obernberger <joseph.obernber...@gmail.com>
> wrote:
>
> Thank you Jeff - yes, this is on the latest 4.0.1
>
> nodetool version
> ReleaseVersion: 4.0.1
> nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 172.16.100.251 488.38 GiB 200 38.7%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN 172.16.100.208 76.02 GiB 30 5.8%
> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
> UN 172.16.100.252 501.88 GiB 200 38.6%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN 172.16.100.249 517.27 GiB 200 38.6%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN 172.16.100.36 524.45 GiB 200 38.6%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN 172.16.100.39 521.05 GiB 200 38.6%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN 172.16.100.253 11.39 GiB 4 0.8%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN 172.16.100.248 524.46 GiB 200 38.7%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN 172.16.100.37 314.67 GiB 120 23.2%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN 172.16.100.250 464.23 GiB 200 38.6%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> yum list installed | grep cass
> cassandra.noarch 4.0.1-1
> @cassandra
>
> -Joe
> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>
> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
> large for the urgent message queue, causing this stack trace, and was
> released 3 days ago. I've never seen it on a 10 node cluster before, but
> I'd be trying that.
>
> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
>> I have a 10 node cluster and am trying to add another node. The new
>> node is running Rocky Linux and I'm getting the unable to gossip with
>> any peers error. Firewall and SELinux are off. I can ping all the
>> other nodes OK. I've checked everything I can think of (/etc/hosts,
>> listen_address, broadcast etc..). It all looks correct to me.
>> Any ideas? Could it be an incompatibility with Rocky?
>>
>> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
>> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
>> <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864
>>
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
>> - Updating topology for all endpoints that have changed
>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>> recording size estimates
>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>> Exception encountered during startup
>> java.lang.RuntimeException: Unable to gossip with any peers
>> at
>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>> at
>>
>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>> at
>>
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>> StorageService.java:1621 - DRAINING: starting drain process
>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>> HintsService.java:220 - Paused hints dispatch
>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>> Gossiper.java:1993 - No local state, state is in silent shutdown, or
>> node hasn't joined, not announcing shutdown
>>
>> Thank you!
>>
>> -Joe
>>
>>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
