[ https://issues.apache.org/jira/browse/CASSANDRA-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249053#comment-14249053 ]
Pavel Yaskevich commented on CASSANDRA-7275: -------------------------------------------- I understand it might be hard for you, Benedict, but just consider there could be a programming error in the flush of the memtable or replacing flushed one, which is only triggered when metadata about compaction is written back at the end of that compaction e.g. CompactionTask.runMayThrow() L225, e.g. error mentioned in the description or "duplicate hard-link failure" or something similar which has nothing to do with the underlaying (file-)system which means that #1 suggestion is not going to help because compaction is blocked in SystemKeyspace.finishCompaction() and flush retry is not going to help because it will just fail again and again trying to flush the same data. As an end user I would prefer that nobody actually takes a decision to fail on the floor for me except me because it means data loss even when problem is not affecting actual write/read path, I would be fine though to fail on FS\{Read, Write\}Error if user explicitly sets it to fail on I/O errors (e.g. "disk_failure_policy", it is like of your #2 but not exactly) otherwise I would rather get notified in the log and carry on so I can take informed decision on my next actions. bq. Eventually, if it cannot recover safely, it should die though, as there will need to be some operator involvement and the reality is not everybody monitors their log files. I'm going to ignore this argument until you actually have experience of running Cassandra in production, otherwise it's the same as talking to the wall. bq. I'm much more comfortable with "things die if something goes catastrophically wrong" than "things start returning nonsense on reads" which is what happens if we mark something flushed that actually wasn't. I remember it was already the same when the disk is full in the DSE, did people actually have fun restoring cluster after it went completely dark? I'm also *not* saying that we shouldn't fail on FS\{Read, Write\}Error if "disk_failure_policy" says otherwise. > Errors in FlushRunnable may leave threads hung > ---------------------------------------------- > > Key: CASSANDRA-7275 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7275 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Tyler Hobbs > Assignee: Pavel Yaskevich > Priority: Minor > Fix For: 2.0.12 > > Attachments: 0001-Move-latch.countDown-into-finally-block.patch, > 7252-2.0-v2.txt, CASSANDRA-7275-flush-info.patch > > > In Memtable.FlushRunnable, the CountDownLatch will never be counted down if > there are errors, which results in hanging any threads that are waiting for > the flush to complete. For example, an error like this causes the problem: > {noformat} > ERROR [FlushWriter:474] 2014-05-20 12:10:31,137 CassandraDaemon.java (line > 198) Exception in thread Thread[FlushWriter:474,5,main] > java.lang.IllegalArgumentException > at java.nio.Buffer.position(Unknown Source) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:64) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:138) > at > org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103) > at > org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:439) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:397) > at > org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)