[ https://issues.apache.org/jira/browse/CASSANDRA-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700392#comment-17700392 ]
Adriano Bonacin commented on CASSANDRA-18153: --------------------------------------------- [~jlewandowski] I have two nodes: {{[root@ip-20-0-1-182 ~]# nodetool status}} {{Datacenter: datacenter1}} {{=======================}} {{Status=Up/Down}} {{|/ State=Normal/Leaving/Joining/Moving}} {{-- Address Load Tokens Owns (effective) Host ID Rack}} {{UN 20.0.1.182 66.2 KiB 2 100.0% 1b0c03de-e885-4028-99c9-24122146f07d rack1}} {{UN 20.0.1.239 66.2 KiB 2 100.0% c54cf0ab-cf6e-46ce-9ebe-86b254fe5606 rack1}} Creating a keyspace + table and writing something (wait to commitlog be flushed to disk if commitlog_sync is periodic): {{[root@ip-20-0-1-182 ~]# cqlsh $HOSTNAME}} {{Connected to AnsibleLab at ip-20-0-1-182.ec2.internal:9042.}} {{[cqlsh 5.0.1 | Cassandra 3.11.13 | CQL spec 3.4.4 | Native protocol v4]}} {{Use HELP for help.}} {{cqlsh> CREATE KEYSPACE abonacin WITH replication = \{'class': 'NetworkTopologyStrategy', 'datacenter1': 2};}} {{cqlsh> CREATE TABLE abonacin.emp (}} {{... emp_id int PRIMARY KEY,}} {{... emp_city text,}} {{... emp_name text,}} {{... emp_phone varint,}} {{... emp_sal varint,}} {{... emp_hiredate date}} {{... );}} {{cqlsh> insert into abonacin.emp (emp_id, emp_city, emp_name, emp_phone, emp_sal, emp_hiredate) values (1584, 'KVXKAAOHYH', 'KLEMTXEDBDIF', 60457532, 1000, '2022-08-25');}} {{cqlsh> select * from abonacin.emp;}} {{ emp_id | emp_city | emp_hiredate | emp_name | emp_phone | emp_sal}} {{--------+------------+--------------+--------------+-----------+---------}} {{ 1584 | KVXKAAOHYH | 2022-08-25 | KLEMTXEDBDIF | 60457532 | 1000}} Now I'll kill -9 java process on the other node and restart cassandra service: {{[root@ip-20-0-1-239 ~]# tail -f /var/log/cassandra/debug.log | grep "Ignored commitLogIntervals from the following sstables" | grep abonacin &}} {{[root@ip-20-0-1-239 ~]# ps -ef | grep -v grep | grep java | awk '\{print $2}' | xargs kill -9}}{{[root@ip-20-0-1-239 ~]# systemctl restart cassandra}} And I see this: {{[root@ip-20-0-1-239 ~]# DEBUG [main] 2023-03-14 21:25:46,486 CommitLogReplayer.java:254 - Ignored commitLogIntervals from the following sstables: [/var/lib/cassandra/data/abonacin/emp-ee4bec70c2ac11ed85b715ea429a0971/me-1-big-Data.db]}} What I found is that Storage Service is not yet available during CommigLogReplay (not sure if it is always starts after commitlog replay or it's just a coincidence). The following +StorageService.instance.getLocalHostUUID()+ returns {_}null{_}. {{private static UUID getLocalHostId()}} {{ {}} {{ return StorageService.instance.getLocalHostUUID();}} {{ }}} I also find that{{ +Optional.ofNullable(StorageService.instance.getLocalHostUUID()).orElseGet(SystemKeyspace::getLocalHostId)+}} returns the correct host uuid. This why I chose this approach instead of yours. It's also worth to mention that it's the first time I tried to suggest a PR and I don't have enough familiarity with all the context involved. :) > Memtable being flushed without hostId in version "me" and newer during > CommitLogReplay > -------------------------------------------------------------------------------------- > > Key: CASSANDRA-18153 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18153 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable > Reporter: Adriano Bonacin > Assignee: Adriano Bonacin > Priority: Normal > Time Spent: 50m > Remaining Estimate: 0h > > On ticket CASSANDRA-16619 some files were changed to allow Cassandra to store > HostID in the new "me" SSTable version. > But SSTables flushed during CommitLogReplay miss this HostID info. > > In the next Cassandra startup, if these SSTables were still present, > system.log will show: > {{WARN Origin of 3 sstables is unknown or doesn't match the local node; > commitLogIntervals for them were ignored}} > {{WARN }}{{{}Origin of 3 sstables is unknown or doesn't match the local node; > commitLogIntervals for them were ignored{}}}{{{}{}}}{{ }} > > And debug.log will show a list of SSTables, witch can include "md" and "me" > version (before upgradesstables): > > {{Ignored commitLogIntervals from the following sstables: > [/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/me-3-big-Data.db, > > /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-1-big-Data.db, > > /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-2-big-Data.db]}} > > https://issues.apache.org/jira/browse/CASSANDRA-16619 -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org