But in your architecture you are going to write the contents of the memory channel out? Or did I miss something?
"The checkpoint will be updated each time we perform a successive insertion into the memory channel." On Tue, Nov 6, 2012 at 3:43 PM, Rahul Ravindran <rahu...@yahoo.com> wrote: > We have a legacy system which writes events to a file (existing log file). > This will continue. If I used a filechannel, I will be double the number of > IO operations(writes to the legacy log file, and writes to WAL). > > ________________________________ > From: Brock Noland <br...@cloudera.com> > To: user@flume.apache.org; Rahul Ravindran <rahu...@yahoo.com> > Sent: Tuesday, November 6, 2012 1:38 PM > Subject: Re: Guarantees of the memory channel for delivering to sink > > Your still going to be writing out all events, no? So how would file > channel do more IO than that? > > On Tue, Nov 6, 2012 at 3:32 PM, Rahul Ravindran <rahu...@yahoo.com> wrote: >> Hi, >> I am very new to Flume and we are hoping to use it for our log >> aggregation into HDFS. I have a few questions below: >> >> FileChannel will double our disk IO, which will affect IO performance on >> certain performance sensitive machines. Hence, I was hoping to write a >> custom Flume source which will use a memory channel, and which will >> perform >> checkpointing. The checkpoint will be updated each time we perform a >> successive insertion into the memory channel. (I realize that this results >> in a risk of data, the maximum size of which is the capacity of the memory >> channel). >> >> As long as there is capacity in the memory channel buffers, does the >> memory channel guarantee delivery to a sink (does it wait for >> acknowledgements, and retry failed packets)? This would mean that we need >> to >> ensure that we do not exceed the channel capacity. >> >> I am writing a custom source which will use the memory channel, and which >> will catch a ChannelException to identify any channel capacity issues(so, >> buffer used in the memory channel is full because of lagging sinks/network >> issues etc). Is that a reasonable assumption to make? >> >> Thanks, >> ~Rahul. > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
