[
https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408429#comment-13408429
]
Patrick Wendell commented on FLUME-1339:
----------------------------------------
Hey Hari - if someone just used an agent with a memory channel and an avro
sink, wouldn't this enact fail-retry semantics? If an avro sink has a problem,
it fails the txn and then attempts to reconnect, so this is a retry - right?
If the goal is to have partial durability and get better performance - one
obvious extension would be to add some type of batching to the FileChannel, so
that it doesn't sync at every txn, but there are sync points periodically. A
disk seek takes 5 ms, so at a rate of 300 events per second you're looking at a
lot of seek time if the disk is being used for other stuff than the flume logs.
If you sync periodically, the data loss is bounded by the sync interval size.
Anyways a clear understanding of the performance bottleneck we are trying to
solve would help navigate towards the correct solution. I guess maybe we can
defer to Juhandi for that.
> Implement an exec source like client API
> ----------------------------------------
>
> Key: FLUME-1339
> URL: https://issues.apache.org/jira/browse/FLUME-1339
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Patrick Wendell
>
> Originally when the exec source was implemented there was no SDK. It would be
> useful to have an API provided in the sdk that works like the exec source,
> thus removing the requirement of having an entire agent on hosts where the
> exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users
> can use off the shelf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
