[ https://issues.apache.org/jira/browse/CASSANDRA-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-17996: ----------------------------------------- Resolution: Invalid Status: Resolved (was: Triage Needed) This issue tracker is for the development of Apache Cassandra and as such makes a poor vehicle for support. I recommend contacting the community through the mailing or slack for your issue: https://cassandra.apache.org/_/community.html > Authentication timed out > ------------------------ > > Key: CASSANDRA-17996 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17996 > Project: Cassandra > Issue Type: Bug > Reporter: Andy Lage > Priority: Normal > > Hi, > > I'm having some issues with my authentication when trying to login with > cqlsh. I'm running a cluster of 3 nodes that is spaced out on three different > physical nodes in Kubernetes. It has been rolling like a charm for the last > month but around a week or so, it started to go down. Down below you can see > when I try to login to the different nodes and the responses. (a FYI it's not > always node 0 that has problem, I've also seen node 1 has the same problem > but then node 0 worked fine. > {code:java} > I have no name!@cassandra-0:/$ cqlsh -u cassandra -p abc123 > Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': > OperationTimedOut('errors=Timed out creating connection (5 seconds), > last_host=None',)}) > I have no name!@cassandra-1:/$ cqlsh -u cassandra -p abc123 > Python 2.7 support is deprecated. Install Python 3.6+ or set > CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at > 127.0.0.1:9042 > [cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5] > Use HELP for help. > cassandra@cqlsh> > I have no name!@cassandra-2:/$ cqlsh -u cassandra -p abc123 > Python 2.7 support is deprecated. Install Python 3.6+ or set > CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at > 127.0.0.1:9042 > [cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5] > Use HELP for help. > cassandra@cqlsh>{code} > Here is the logs from node 0 > > {code:java} > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - > HINT_RSP messages were dropped in last 5000 ms: 0 internal and 4 cross node. > Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 > ms > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - > HINT_REQ messages were dropped in last 5000 ms: 0 internal and 2 cross node. > Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 > ms > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:65 - Pool > Name Active Pending Completed Blocked All > Time Blocked > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - > ReadStage 0 0 17086 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - > CompactionExecutor 0 0 195522 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - > MutationStage 0 0 88515 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > MemtableReclaimMemory 0 0 210 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > PendingRangeCalculator 0 0 13 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > GossipStage 0 0 660680 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > SecondaryIndexManagement 0 0 0 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > HintsDispatcher 2 0 37776 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > Repair-Task 0 0 5 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > Native-Transport-Requests 0 0 17111 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > RequestResponseStage 0 0 20 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > MemtableFlushWriter 0 0 210 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > PerDiskMemtableFlushWriter_0 0 0 210 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > MemtablePostFlush 0 0 255 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > Sampler 0 0 0 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > ValidationExecutor 0 0 41 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > ViewBuildExecutor 0 0 0 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - > InternalResponseStage 0 0 151676 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - > AntiEntropyStage 0 0 249 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - > CacheCleanupExecutor 0 0 0 0 > 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:79 - > CompactionManager 0 0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:91 - > MessagingService n/a 0/0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:101 - > Cache Type Size Capacity > KeysToSave > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:103 - > KeyCache 11452 75497472 > all > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:109 - > RowCache 0 0 > all > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:116 - > Table Memtable ops,data > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:119 - > system_schema.columns 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.types 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.indexes 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.keyspaces 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.dropped_columns 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.aggregates 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.triggers 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.tables 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.views 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system_schema.functions 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.compaction_history 3,634 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.IndexInfo 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.repairs 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.size_estimates 49344,1087744 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.table_estimates 98688,2339296 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.paxos 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.built_views 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.peer_events 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.peers_v2 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.peers 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.peer_events_v2 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.batches 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - > system.transferred_ranges 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.transferred_ranges_v2 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.view_builds_in_progress 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.local 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.sstable_activity 229,2816 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.available_ranges_v2 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.available_ranges 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - > system.prepared_statements 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - > system_auth.roles 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - > system_auth.role_members 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - > system_auth.resource_role_permissons_index 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - > system_auth.network_permissions 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - > system_auth.role_permissions 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - > system_distributed.parent_repair_history 10,418916 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - > system_distributed.repair_history 29495,15967 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - > system_distributed.view_build_status 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - > system_traces.sessions 0,0 > INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - > system_traces.events 0,0 > {code} > The keyspace replication > > {code:java} > cassandra@cqlsh> describe keyspace system_auth; > CREATE KEYSPACE system_auth WITH replication = {'class': > 'NetworkTopologyStrategy', 'datacenter1': '3'} AND durable_writes = true; > {code} > Nodetool status > {code:java} > I have no name!@cassandra-2:/$ nodetool status Datacenter: datacenter1 > ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID Rack UN 10.233.92.33 3.63 MiB > 256 100.0% xxx rack1 UN 10.233.96.184 4.2 MiB 256 100.0% xxx rack1 UN > 10.233.90.48 3.24 MiB 256 100.0% xxx rack1 {code} > I've tried to do nodetool repair without success. > > Does anyone have a clue whats going on? -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org