How large is your heap? You will likely want two data directories per disk. Also with a channel that large I strongly recommend using back up checkpoints.
Additionally https://issues.apache.org/jira/browse/FLUME-2155 will be very useful to you as well. On Jan 30, 2014 4:21 AM, "Umesh Telang" <[email protected]> wrote: > Hi Hari, > > The capacity of the channel is 150,000,000. The other properties of the > file channel are as below: > a1.channels.s3-file-channel.type = file > a1.channels.s3-file-channel.checkpointDir = > /mnt/flume-file-channels/s3-file-channel/checkpoint > a1.channels.s3-file-channel.dataDirs = > /mnt/flume-file-channels/s3-file-channel/data > a1.channels.s3-file-channel.transactionCapacity = 20000 > a1.channels.s3-file-channel.capacity = 150000000 > > We've been experimenting with the configuration. We haven't specifically > noticed an increase in the checkpoint size. It's just that as the size > we've observed is in the order of gigabytes, we wanted to understand how > the checkpoint size would vary, if at all. > > Based on what you've said, it looks like the checkpoint size is a direct > function of the channel capacity. So, for a given channel capacity... as > long as there is enough disk space initially provisioned, that should be > sufficient for that flume agent. > > Thanks again for clarifying! > > Umesh > > > ------------------------------ > *From:* Hari Shreedharan [[email protected]] > *Sent:* 29 January 2014 18:55 > *To:* [email protected] > *Subject:* Re: checkpoint lifecycle > > What is the capacity of your channel? I would assume that the > checkpoint size will remain the same throughout. > > > Thanks, > Hari > > On Wednesday, January 29, 2014 at 9:37 AM, Umesh Telang wrote: > > Thanks for the quick response, Hari! > > We are using version 1.4.0 of Flume. > > The contents and sizes of the checkpoint directory are as below: > $ ls -lh > total 1.2G > -rw-r--r-- 1 flume flume 1.2G Jan 29 17:34 checkpoint > -rw-r--r-- 1 flume flume 25 Jan 29 17:34 checkpoint.meta > -rw-r--r-- 1 flume flume 0 Jan 28 07:56 in_use.lock > -rw-r--r-- 1 flume flume 32 Jan 29 17:34 inflightputs > -rw-r--r-- 1 flume flume 32 Jan 29 17:34 inflighttakes > > Thanks, > Umesh > > > ------------------------------ > *From:* Hari Shreedharan [[email protected]] > *Sent:* 29 January 2014 17:22 > *To:* [email protected] > *Subject:* Re: checkpoint lifecycle > > The checkpoint file itself should be fixed size though other files in > that directory may vary in size. What version of flume are you using? Newer > versions should have more files in that directory. > > On Wednesday, January 29, 2014, Umesh Telang <[email protected]> > wrote: > > Hello, > > Under a file channels checkpoint directory, I see the following files: > checkpoint > checkpoint.meta > > I wanted to know whether the size of the checkpoint file should reach a > steady state if the amount and rate of input to the file chain remains the > same. > > My understanding is that the checkpoint file is associate with the write > ahead log. Is this something that continues to grow indefinitely? > > Or is there some lifecycle management that cleans out very old entries > from the write ahead log? > > If not, is there some strategy that we should employ to manage the size > of the checkpoint file (in our case, it's currently over 1GB after 2 days' > operation). > > Thanks for any advice on this. > > Kind regards, > Umesh > > > > ---------------------------- > > http://www.bbc.co.uk > This e-mail (and any attachments) is confidential and may contain personal > views which are not the views of the BBC unless specifically stated. > If you have received it in error, please delete it from your system. > Do not use, copy or disclose the information in any way nor act in > reliance on it and notify the sender immediately. > Please note that the BBC monitors e-mails sent or received. > Further communication will signify your consent to this. > > --------------------- > > > > ---------------------------- > > http://www.bbc.co.uk > This e-mail (and any attachments) is confidential and may contain personal > views which are not the views of the BBC unless specifically stated. > If you have received it in error, please delete it from your system. > Do not use, copy or disclose the information in any way nor act in > reliance on it and notify the sender immediately. > Please note that the BBC monitors e-mails sent or received. > Further communication will signify your consent to this. > > --------------------- > > > > > ---------------------------- > > http://www.bbc.co.uk > This e-mail (and any attachments) is confidential and may contain personal > views which are not the views of the BBC unless specifically stated. > If you have received it in error, please delete it from your system. > Do not use, copy or disclose the information in any way nor act in > reliance on it and notify the sender immediately. > Please note that the BBC monitors e-mails sent or received. > Further communication will signify your consent to this. > > --------------------- >
