Jay Kreps created KAFKA-615:
-------------------------------
Summary: Avoid fsync on log segment roll
Key: KAFKA-615
URL: https://issues.apache.org/jira/browse/KAFKA-615
Project: Kafka
Issue Type: Bug
Reporter: Jay Kreps
Assignee: Neha Narkhede
It still isn't feasible to run without an application level fsync policy. This
is a problem as fsync locks the file and tuning such a policy so that the
flushes aren't so frequent that seeks reduce throughput, yet not so infrequent
that the fsync is writing so much data that there is a noticable jump in
latency is very challenging.
The remaining problem is the way that log recovery works. Our current policy is
that if a clean shutdown occurs we do no recovery. If an unclean shutdown
occurs we recovery the last segment of all logs. To make this correct we need
to ensure that each segment is fsync'd before we create a new segment. Hence
the fsync during roll.
Obviously if the fsync during roll is the only time fsync occurs then it will
potentially write out the entire segment which for a 1GB segment at 50mb/sec
might take many seconds. The goal of this JIRA is to eliminate this and make it
possible to run with no application-level fsyncs at all, depending entirely on
replication and background writeback for durability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
