[ https://issues.apache.org/jira/browse/CASSANDRA-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Hanna updated CASSANDRA-11714: ------------------------------------- Labels: lhf (was: ) > shutdown script should wait more than 30 seconds before force kill > ------------------------------------------------------------------ > > Key: CASSANDRA-11714 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11714 > Project: Cassandra > Issue Type: Improvement > Components: Tools > Reporter: Wei Deng > Priority: Minor > Labels: lhf > > I'm running with 3.0.6 and it appears that if I let it run for multiple days > (ingesting a never-ending streaming data from Kafka), then the next time I'm > restarting the JVM by "service cassandra restart", it will take 9 minutes to > replay the commit log segments (the commitlog_total_space_in_mb is using the > default 8GB and I observed close to 8GB commitlog segment files before the > restart). Here is a fragment of the system.log showing how long it takes: > {noformat} > INFO [main] 2016-05-04 19:06:54,356 CommitLog.java:168 - Replaying > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248425.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248429.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248433.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248437.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248441.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248445.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248449.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248453.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248457.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248461.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248466.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248470.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248474.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248478.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248482.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248486.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248490.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248494.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248498.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248502.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248506.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248510.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248514.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248518.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248522.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248526.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248528.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248529.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248530.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248531.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248532.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248533.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248534.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248535.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248536.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248537.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248538.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248539.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248540.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248541.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248542.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248543.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248544.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248545.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248546.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248547.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248548.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248549.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248550.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248551.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248552.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248553.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248554.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248555.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248556.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248557.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248558.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248559.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248560.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248561.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248562.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248563.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248564.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248565.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248566.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248567.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248568.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248569.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248570.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248571.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248572.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248573.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248574.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248575.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248576.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248577.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248578.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248579.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248580.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248581.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248582.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248583.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248584.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248585.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248586.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248587.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248588.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248589.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248590.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248591.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248592.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248593.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248594.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248595.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248596.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248597.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248598.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248599.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248600.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248601.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248602.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248603.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248604.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248605.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248606.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248607.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248608.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248609.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248610.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248611.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248612.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248613.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248614.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248615.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248616.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248617.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248618.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248619.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248620.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248621.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248622.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248623.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248624.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248625.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248626.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248627.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248628.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248629.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248630.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248631.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248632.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248633.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248634.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248635.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248636.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248637.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248638.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248639.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248640.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248641.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248642.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248643.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248644.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248645.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248646.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248647.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248648.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248649.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248650.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248651.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248652.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248653.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248654.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248655.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248656.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248657.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248658.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248659.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248660.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248661.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248662.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248663.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248664.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248665.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248666.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248667.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248668.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248669.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248670.log, > /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248671.log > INFO [main] 2016-05-04 19:15:44,172 CommitLog.java:170 - Log replay > complete, 331463 replayed mutations > {noformat} > It appears that the init script here only gives 30-second grace period after > SIGTERM before forcefully killing it. > https://github.com/apache/cassandra/blob/cassandra-3.0/debian/init/#L101 > If we could increase this 30-second to a higher wait time like 1 or 2 minutes > we could make the startup delay much shorter. > Keep in mind that we don't always have to wait for this long during the > shutdown so we're not really losing anything with this improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)