Hi Phil, thanks for your reply.
Yes, setting up a file-channel configuration is consuming CPU up to 80/90% My settings: # Channel configuration agent1.channels.ch1.type = file agent1.channels.ch1.checkpointDir = /opt/flume-ng/chekpoint agent1.channels.ch1.dataDirs = /opt/flume-ng/data agent1.channels.ch1.capacity = 1000000 agent1.channels.ch1.transactionCapacity = 10000 # flume-env.sh export JAVA_OPTS="-Xms512m -Xmx2048m" # top 22079 flume-ng 20 0 6924752 785536 17132 S 83.7% 2.4 3:53.19 java Do you have any tuning for the GC ? Thanks Simone On Thu, Oct 15, 2015 at 7:59 PM, Phil Scala <[email protected]> wrote: > Hi Simone > > > > I wonder why you’re seeing 90% CPU use when you use a file channel. I > would expect high disk I/O. To counter, I have on a single server 4 spool > dir sources, each going to a separate file channel. Also on an SSD based > server. I do not see any CPU or even disk IO utilization. I am pushing > about 10 million events per day across all 4 sources and has been running > reliably for 2 years now. > > > > I would always use a file channel, any memory channel runs the risk of > data loss if the node were to fail. I would be as worried about the local > node failing seeing that a 3 node kafka cluster losing 2 nodes before it > would lose quorum. > > > > Not sure what your data source is, if you can add more flume nodes of > course that would help. > > > > Have you given ample heap space, seeing maybe GC’s causing the high CPU? > > > > > > Phil > > > > > > *From:* Simone Roselli [mailto:[email protected]] > *Sent:* Friday, October 09, 2015 12:33 AM > *To:* [email protected] > *Subject:* Flume-ng 1.6 reliable setup > > > > Hi, > > > > I'm currently plan to migrate from Flume 0.9 to Flume-ng 1.6, but I'm > having troubles trying to find a reliable setup for this one. > > > > My sink is a 3 nodes Kafka cluster. I must avoid *to lose events in case > the main sink is down*, broken or unreachable for a while. > > > > In Flume 0.9, I use a memory channel with the *store on failure *feature, > which starts writing events on the local disk in case the target sink is > not available. > > > > In Flume-ng 1.6 the same behaviour would be accomplished by setting up a > *Spillable > memory channel, *but the problem with this solution is written in the end > of the channel's description: "*This channel is currently experimental > and not recommended for use in production."* > > > > In Flume-ng 1.6, it's possible to setup a pool of *Failover sinks*. So, I > was thinking to hypothetically configure a *File Roll *as Secondary sink > in case the Primary is down. However, once the Primary sink would be back > online, the data placed on the Secondary sink (local disk) won't be > automatically pushed on the Primary one. > > > > Another option would be setting up a *file channel*: write each event on > the disk and then sink. Without mentioning that I don't love the idea to > write/delete each single event continuously on a SSD, this setup is taking > 90% of CPU. The same exactly configuration but using a memory channel takes > 3%. > > > > Other solutions to evaluate ? > > > > Simone > > >
