1 - Bump up the -Xmx in flume-env.sh as the default is quite small 2 - increase the capacity on the channel. Looks like your source is running much faster than the sink can keep up. You can try adding more sinks to improve drain rate.
From: IT CTO <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, October 8, 2015 9:46 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Flume source channel sink tunning Hi, I am using SpoolDir with memory channel to write to hdfs sink. When I use a single spoolDir I get single threaded performance so based on some mails I read I splinted the source to 5 spoolDir all writing to the same memory channel which writes to the hdfs. Now I am getting different errors 1) GC error => not engough memory for the channel => increase Xmx for the agent 2) Channel is full => the sink is not kipping up with the channel so I find myself playing with the different parameters. any best practice here or path to follow to get it tuned? I feel that even if it works it will easily break given other events Eran -- Eran | "You don't need eyes to see, you need vision" (Faithless)
