[ https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187136#comment-16187136 ]
Jeff Jirsa commented on CASSANDRA-13123: ---------------------------------------- I noticed it on 3.0 branch, I haven't had time to investigate but I suspect it may be a test ordering issue (if the two tests are run in one order they pass, in the other they fail, so probably setup/teardown conditions). The first failure I see in cassci (datastax's CI environment, which I don't have access to other than the public read-only view) is http://cassci.datastax.com/job/cassandra-3.0_testall/954/ , which is the build after this change was committed ( http://cassci.datastax.com/job/cassandra-3.0_testall/953/ ) . It also fails in: http://cassci.datastax.com/job/cassandra-3.0_testall/964/ http://cassci.datastax.com/job/cassandra-3.0_testall/963/ http://cassci.datastax.com/job/cassandra-3.0_testall/956/ So % wise, it seems like 4 failures in the 15 builds since introduction. > Draining a node might fail to delete all inactive commitlogs > ------------------------------------------------------------ > > Key: CASSANDRA-13123 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13123 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Reporter: Jan Urbański > Assignee: Jan Urbański > Fix For: 3.0.15, 3.11.1, 4.0 > > Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, > 13123-trunk.txt > > > After issuing a drain command, it's possible that not all of the inactive > commitlogs are removed. > The drain command shuts down the CommitLog instance, which in turn shuts down > the CommitLogSegmentManager. This has the effect of discarding any pending > management tasks it might have, like the removal of inactive commitlogs. > This in turn leads to an excessive amount of commitlogs being left behind > after a drain and a lengthy recovery after a restart. With a fleet of dozens > of nodes, each of them leaving several GB of commitlogs after a drain and > taking up to two minutes to recover them on restart, the additional time > required to restart the entire fleet becomes noticeable. > This problem is not present in 3.x or trunk because of the CLSM rewrite done > in CASSANDRA-8844. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org