[ https://issues.apache.org/jira/browse/CASSANDRA-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543446#comment-13543446 ]
Michael Kjellman commented on CASSANDRA-4446: --------------------------------------------- did a nodetool drain before 1.1.7 -> 1.2.0. upon starting 1.2.0 every node in my cluster still replayed the commit logs and created mutations. {code} INFO 13:17:06,529 DRAINING: starting drain process INFO 13:17:06,529 Stop listening to thrift clients INFO 13:17:06,532 Announcing shutdown INFO 13:17:07,536 Waiting for messaging service to quiesce INFO 13:17:07,537 MessagingService shutting down server thread. ... normal startup stuff.. INFO 13:20:20,182 Replaying /ssd/commitlog/CommitLog-1355265349912.log INFO 13:20:24,166 Finished reading /ssd/commitlog/CommitLog-1355265349912.log INFO 13:20:24,166 Replaying /ssd/commitlog/CommitLog-1355265349914.log INFO 13:20:26,700 Finished reading /ssd/commitlog/CommitLog-1355265349914.log INFO 13:20:26,701 Replaying /ssd/commitlog/CommitLog-1355265349915.log INFO 13:20:28,118 Finished reading /ssd/commitlog/CommitLog-1355265349915.log ... more replay lines ... INFO 13:22:00,061 Log replay complete, 8052 replayed mutations INFO 13:22:00,358 Possible old-format hints found. Truncating INFO 13:22:00,370 Enqueuing flush of Memtable-local@1908923620(402/402 serialized/live bytes, 13 ops) INFO 13:22:00,372 Writing Memtable-local@1908923620(402/402 serialized/live bytes, 13 ops) INFO 13:22:00,494 Cassandra version: 1.2.0 INFO 13:22:00,495 Thrift API version: 19.35.0 INFO 13:22:00,495 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0) INFO 13:22:00,534 Loading persisted ring state INFO 13:22:00,537 Starting up server gossip WARN 13:22:00,557 No host ID found, created dd3a40e2-fef1-4574-87b8-e2929fd80235 (Note: This should happen exactly once per node). {code} > nodetool drain sometimes doesn't mark commitlog fully flushed > ------------------------------------------------------------- > > Key: CASSANDRA-4446 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4446 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.0.10 > Environment: ubuntu 10.04 64bit > Linux HOSTNAME 2.6.32-345-ec2 #48-Ubuntu SMP Wed May 2 19:29:55 UTC 2012 > x86_64 GNU/Linux > sun JVM > cassandra 1.0.10 installed from apache deb > Reporter: Robert Coli > Attachments: > cassandra.1.0.10.replaying.log.after.exception.during.drain.txt > > > I recently wiped a customer's QA cluster. I drained each node and verified > that they were drained. When I restarted the nodes, I saw the commitlog > replay create a memtable and then flush it. I have attached a sanitized log > snippet from a representative node at the time. > It appears to show the following : > 1) Drain begins > 2) Drain triggers flush > 3) Flush triggers compaction > 4) StorageService logs DRAINED message > 5) compaction thread excepts > 6) on restart, same CF creates a memtable > 7) and then flushes it [1] > The columnfamily involved in the replay in 7) is the CF for which the > compaction thread excepted in 5). This seems to suggest a timing issue > whereby the exception in 5) prevents the flush in 3) from marking all the > segments flushed, causing them to replay after restart. > In case it might be relevant, I did an online change of compaction strategy > from Leveled to SizeTiered during the uptime period preceding this drain. > [1] Isn't commitlog replay not supposed to automatically trigger a flush in > modern cassandra? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira