Our current understanding is that there is synchronization in linux so the underlying fsync and write calls actually block each other so even if you have a separate thread do the flush it doesn't help. We would be interested in finding a way to work around this.
Good blog on this: http://antirez.com/post/fsync-different-thread-useless.html -Jay On Wed, May 9, 2012 at 9:35 PM, S Ahmed <sahmed1...@gmail.com> wrote: > couldn't this be written in a way that it doesn't block? or limits the > time, like it makes a copy of the reference, then replaces it when a newed > up channel (synchronized). > > Oh, this isn't possible because the var is mapped to a file at the o/s > level? > > On Tue, May 8, 2012 at 5:38 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> filechannel.force() always fully syncs the file to disk. This is done >> irrespective of message boundaries. The file is locked during this >> time so other appends are blocked. >> >> -Jay >> >> On Tue, May 8, 2012 at 1:44 PM, S Ahmed <sahmed1...@gmail.com> wrote: >> > Slowly trying to understand it, have to wramp up on my scala. >> > >> > When the flush/sink occurrs, does it pull items of the collection 1 by 1 >> or >> > does it do this in bulk somehow while locking the collection? >> > >> > On Mon, May 7, 2012 at 3:14 PM, Neha Narkhede <neha.narkh...@gmail.com >> >wrote: >> > >> >> Ahmed, >> >> >> >> The related code is in kafka.log.*. The message to file persistence is >> >> inside FileMessageSet.scala. >> >> >> >> Thanks, >> >> Neha >> >> >> >> On Mon, May 7, 2012 at 12:12 PM, S Ahmed <sahmed1...@gmail.com> wrote: >> >> >> >> > I can barely read scala, but I'm curious where the applications >> performs >> >> > the operation of taking the in-memory log and persisting it to the >> >> > database, all the while accepting new log messages and removing the >> keys >> >> of >> >> > the messages that have been persisted to disk. >> >> > >> >> > I'm guessing you have used the concurrenthashmap where the key is a >> >> topic, >> >> > and once the flush timeout has been reached a background thread will >> >> > somehow persist and remove the keys. >> >> > >> >> >>
