RE: Perfromacnce issue in file channel

[mahendran m]

Hi Hari ,
Thanks , yes i am sending the event one by one only as it is my requirement but 
memory channel can easily process 1000 per second but file channel doesn't is 
this case how can i improve the performance of file channel .
Regards,Mahendran

Date: Mon, 10 Nov 2014 11:34:57 -0800
From: hshreedha...@cloudera.com
To: user@flume.apache.org
CC: user@flume.apache.org
Subject: Re: Perfromacnce issue in file channel


Sending one event by one event is likely to cause performance issues. In 
addition, I have no idea of the performance on Windows. Btw, 100-200 request 
per second is really orders of magnitude lower than what File Channel can 
handle.

Thanks,
Hari


On Sun, Nov 9, 2014 at 7:49 AM, mahendran m <mahendra...@hotmail.com> wrote:




Hi All ,
I have tested the file channel and file channel performance. below the code 
snippet to generate 1000 event per sec 




while (watch.Elapsed.TotalSeconds < sec)
{

        if (counter <= 1000)

        {

                uuid = uuid + 1;



                var text = HttpUtility.UrlEncode("HI , How are you", 
Encoding.UTF8);



                var flumeEvent = new ThriftFlumeEvent();

                flumeEvent.Headers = new Dictionary<string, string>() { { 
"Host", "http://www.bing.com/"; } };

                flumeEvent.Body = Encoding.ASCII.GetBytes(text);

                Client.append(flumeEvent);

                counter += 1;

        }
}



i succeed in sending the 1000 event per sec when using memory channel , but in 
case if i used file channel  i can  only reach upto 40 to 50 events per second


config for memory channel

# Use memory channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 10
a1.channels.c1.byteCapacity = 5368709120





config for file channel



#using the file channel
a1.channels.c1.type = file
a1.channels.c1.capacity = 500000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = 
F:/.checkPoint


Can anyone explain why file channel is this much channel ? . if i used file 
channel in server for directly logging each request is it better option as my 
server receive 100 to 200 request per second


Regards,
Mahendran