> On Aug. 3, 2012, 9:29 a.m., Mike Percy wrote: > > Wow you have been busy, that is really great! One problem with this change > > however is that FileChannel is no longer guaranteed to be durable. Many > > users cannot accept that limitation, especially after a Flume 1.2.0 release > > with a durable File Channel. Why not just use MemoryChannel if you don't > > need crash durability?
Flume doesn't guarantee 100% delivery. The difference between using 'force' and not using is distinguish with process or server crash. The possible or process crash is most frequently than server crash. Even we use 'force' method in transaction commit each time, we always cannot keep reliable delivery. MemoryChannel may cause too many full gc, I'm tracking gc issue now. Maybe I will use DirectByteBuffer to reduce lots of events in heap. - Denny ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6329/#review9814 ----------------------------------------------------------- On Aug. 3, 2012, 9:39 a.m., Denny Ye wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6329/ > ----------------------------------------------------------- > > (Updated Aug. 3, 2012, 9:39 a.m.) > > > Review request for Flume, Hari Shreedharan and Patrick Wendell. > > > Description > ------- > > Here is the description in code changes > 1. Remove the 'FileChannel.force(false)'. Each commit from Source will invoke > this 'force' method. This method is too heavy for amounts of data comes. Each > 'force' action will be consume 50-500ms that it confirms data stored into > disk. Normally, OS will flush data from kernal buffer to disk asynchronously > with ms level latency. It may useless in each commit operation. Certainly, > data loss may occurs in server crash not process crash. Server crash is > infrequent. > 2. Do not pre-allocate disk space. Disk doesn't need the pre-allocation. > 3. Use 'RandomAccessFile.write()' to replace 'FileChannel.write()'. Both in > my test result and low-level instruction, the former is better than the latter > > Here I posted three changes, and I would like to use thread-level cached > DirectByteBuffer to replace inner-heap ByteBuffer.allocate() (reuse > outer-heap memory to reduce time that copying from heap to kernal). I will > test this changes in next phase. > > After tuning, throughput increasing from 5MB to 30MB > > > This addresses bug FLUME-1423. > https://issues.apache.org/jira/browse/FLUME-1423 > > > Diffs > ----- > > > trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java > 1363210 > > Diff: https://reviews.apache.org/r/6329/diff/ > > > Testing > ------- > > > Thanks, > > Denny Ye > >
