[ https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179822#comment-16179822 ]
Bessenyei Balázs Donát commented on FLUME-3149: ----------------------------------------------- Hi [~zyfo2], I'm a little busy nowadays, but I'll do my best to review. I really hope there are other people who can help with the review on the PR. Thank you, Donat PS. What I found on a first pass is that we should probably add more info to the user guide, ie. "the "x" sources and the "y" channels are all compatible / maintain backwards compatibility, but when used in this or that combination, one should be able to achieve reasonably higher performance. > reduce cpu cost for file source transfer while still maintaining reliability > ---------------------------------------------------------------------------- > > Key: FLUME-3149 > URL: https://issues.apache.org/jira/browse/FLUME-3149 > Project: Flume > Issue Type: Improvement > Components: File Channel > Reporter: Will Zhang > Assignee: Will Zhang > > File channel tracks transferred events and use transnational mechanism to > make transfer recoverable. However, it increases CPU cost due to frequent > system calls like write, read, etc. The Cpu cost could be very high if the > transfer rate is high. In contrast, Memory channel has no such issue which > requires only about 10% of CPU cost in the same environment but it's not > recovered if the system is down accidentally. > For sources like taildir/spooldir, I propose we could track offsets of file > and store them locally to achieve reliability while still using memory > channel to reduce CPU cost. Actually, I have already implemented this feature > by storing the offsets in event headers and passing it to my own > "offsetMemoryChannel" and store theses offsets in local disk in our > production which reduces CPU cost by about 90 percent. > Please let me know if it's worthwhile to have this feature in community > version. Thank you. -- This message was sent by Atlassian JIRA (v6.4.14#64029)