[ https://issues.apache.org/jira/browse/CASSANDRA-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054433#comment-14054433 ]
Karl Mueller edited comment on CASSANDRA-7507 at 7/8/14 2:23 AM: ----------------------------------------------------------------- if a bug can cause the clean exit after OOM to fail as expected, then isn't it considered a problem? I guess if I'm considering the value of a "clean exit" versus "possibly staying up, being in a weird state, or not writing the right data to disk", I would always prefer it to die without worrying about a clean exit. As I said, in my opinion, Cassandra already handles dying unexpectedly fine - there's no need to handle it cleanly when there's any risk. If there's no risk of something like 7133 happening (or a similar bug), then sure, clean exit is sensible, but that's clearly not guaranteed. Replaying some logs and then flushing is not a big deal compared to potentially bad data, zombie states, etc. - in my view, at least. was (Author: kmueller): if a bug can cause the clean exit after OOM to fail as expected, then isn't it considered a problem? I guess if I'm considering the value of a "clean exit" versus "possibly staying up or being in a weird state", I would always prefer it to die without worrying about a clean exit. As I said, in my opinion, Cassandra already handles dying unexpectedly fine - there's no need to handle it cleanly when there's any risk. If there's no risk of something like 7133 happening (or a similar bug), then sure, clean exit is sensible, but that's clearly not guaranteed. Replaying some logs and then flushing is not a big deal compared to potentially bad data, zombie states, etc. - in my view, at least. > OOM creates unreliable state - die instantly better > --------------------------------------------------- > > Key: CASSANDRA-7507 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7507 > Project: Cassandra > Issue Type: New Feature > Reporter: Karl Mueller > Priority: Minor > > I had a cassandra node run OOM. My heap had enough headroom, there was just > something which either was a bug or some unfortunate amount of short-term > memory utilization. This resulted in the following error: > WARN [StorageServiceShutdownHook] 2014-06-30 09:38:38,251 StorageProxy.java > (line 1713) Some hints were not written before shutdown. This is not > supposed to happen. You should (a) run repair, and (b) file a bug report > There are no other messages of relevance besides the OOM error about 90 > minutes earlier. > My (limited) understanding of the JVM and Cassandra says that when it goes > OOM, it will attempt to signal cassandra to shut down "cleanly". The problem, > in my view, is that with an OOM situation, nothing is guaranteed anymore. I > believe it's impossible to reliably "cleanly shut down" at this point, and > therefore it's wrong to even try. > Yes, ideally things could be written out, flushed to disk, memory messages > written, other nodes notified, etc. but why is there any reason to believe > any of those steps could happen? Would happen? Couldn't bad data be written > at this point to disk rather than good data? Some network messages delivered, > but not others? > I think Cassandra should have the option to (and possibly default) to kill > itself immediately upon the OOM condition happening in a hard way, and not > rely on the java-based clean shutdown process. Cassandra already handles > recovery from unclean shutdown, and it's not a big deal. My node, for > example, kept in a sort-of alive state for 90 minutes where who knows what it > was doing or not doing. > I don't know enough about the JVM and options for it to know the best exact > implementation of "die instantly on OOM", but it should be something that's > possible either with some flags or a C library (which doesn't rely on java > memory to do something which it may not be able to get!) > Short version: a kill -9 of all C* processes in that instance without needing > more java memory, when OOM is raised -- This message was sent by Atlassian JIRA (v6.2#6252)