It will keep any tx log that has a corresponding event in the channel + 2 per data directory.
On Mon, Nov 25, 2013 at 2:55 PM, Ritesh Adval <[email protected]> wrote: > Thanks but we do not know how many transaction log files it will create, so > it may go out of disk space even if we set lower maxFileSize. Do we know > how many max log files it will keep in flume 1.4 ? > > Ritesh > > > > > On Mon, Nov 25, 2013 at 12:50 PM, Brock Noland <[email protected]> wrote: >> >> Lower the maxFileSize. >> >> On Mon, Nov 25, 2013 at 2:41 PM, Ritesh Adval <[email protected]> >> wrote: >> > Hi, >> > >> > We are running two flume 1.4 agents each with 2 file channel on a VM of >> > size 15GB. >> > >> > Is VM recommded to run flume or do we need bare metal boxes? >> > >> > >> > Every week or so we are running into situation where due to our sinks on >> > these agents not able to send message to upstream agents, the flume file >> > channels get filled with large transaction logs. >> > >> > Here is what we see on 4 channels : >> > >> > $ du -h /srv/flume/ >> > 4.9G /srv/flume/metricChannel1-Cluster/data >> > 7.7M /srv/flume/metricChannel1-Cluster/checkpoint >> > 4.9G /srv/flume/metricChannel1-Cluster >> > 4.9G /srv/flume/metricChannel2-Cluster/data >> > 7.7M /srv/flume/metricChannel2-Cluster/checkpoint >> > 4.9G /srv/flume/metricChannel2-Cluster >> > 214M /srv/flume/eventChannel2-Cluster/data >> > 7.7M /srv/flume/eventChannel2-Cluster/checkpoint >> > 222M /srv/flume/eventChannel2-Cluster >> > 215M /srv/flume/eventChannel1-Cluster/data >> > 7.7M /srv/flume/eventChannel1-Cluster/checkpoint >> > 223M /srv/flume/eventChannel1-Cluster >> > 11G /srv/flume/ >> > >> > >> > Here is an example of tx logs on metricChannel1, we are seeing 5 log >> > files. >> > Is there >> > a way to restrict the number of log files kept? I think in older version >> > of >> > flume it was max 2 log files but we are seeing more than 2 as shown >> > below: >> > >> > >> > $ ls -l /srv/flume/metricChannel1-Cluster/data/ >> > total 4.5G >> > -rw-r--r-- 1 flume flume 0 Nov 23 00:39 in_use.lock >> > -rw-r--r-- 1 flume flume 1.1G Nov 23 11:11 log-1 >> > -rw-r--r-- 1 flume flume 47 Nov 24 21:14 log-1.meta >> > -rw-r--r-- 1 flume flume 1.1G Nov 23 21:18 log-2 >> > -rw-r--r-- 1 flume flume 47 Nov 24 21:14 log-2.meta >> > -rw-r--r-- 1 flume flume 1.1G Nov 24 07:13 log-3 >> > -rw-r--r-- 1 flume flume 47 Nov 24 21:14 log-3.meta >> > -rw-r--r-- 1 flume flume 1.1G Nov 24 17:08 log-4 >> > -rw-r--r-- 1 flume flume 47 Nov 24 21:14 log-4.meta >> > -rw-r--r-- 1 flume flume 425M Nov 24 21:15 log-5 >> > -rw-r--r-- 1 flume flume 47 Nov 24 21:14 log-5.meta >> > >> > >> > we have set maxFileSize to 1GB and it looks like each tx log is within >> > that >> > limit and capacity on file channel to 1M message >> > >> > agent.channels.metricChannel2.transactionCapacity=1000 >> > agent.channels.metricChannel2.capacity=1000000 >> > agent.channels.metricChannel2.maxFileSize=1073741824 >> > >> > >> > What we want to avoid is transaction log filling up the disk, Is there >> > a >> > way to achieve this. >> > We are ok to discard the message. >> > >> > Thanks >> > Ritesh >> > >> > >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org > > -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
