Arkadiusz Firus created KAFKA-2997: -------------------------------------- Summary: Synchronous write to disk Key: KAFKA-2997 URL: https://issues.apache.org/jira/browse/KAFKA-2997 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9.0.0 Reporter: Arkadiusz Firus Priority: Minor
Hi All, I am currently work on a mechanism which allows to do an efficient synchronous writing to the file system. My idea is to gather few write requests for one partition and after that call the fsync. As I read the code I find out that the best place to do it is to modify: kafka.log.Log.append method. Currently at the end of the method (line 368) there is a verification if the number of unflushed messages is greater than the flush interval (configuration parameter). I am thinking of extending this condition. I want to add additional boolean configuration parameter (sync write or something like this). If this parameter is set to true at the end of this method the thread should hang on a lock. On the other hand there will be another timer thread (for every partition) which will be invoked every 10ms (configuration parameter). During invocation the thread will call flush method and after that will be releasing all hanged threads. I am writing here because I would like to know your opinion about such approach. Do you think this one is good or maybe someone have a better (more permanent) one. I would also like to know if such approach is according to general Kafka architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)