kaushik srinivas created FLUME-3211:
---------------------------------------
Summary: Byte capacity seems limiting the flume kafka source
operation.
Key: FLUME-3211
URL: https://issues.apache.org/jira/browse/FLUME-3211
Project: Flume
Issue Type: Question
Reporter: kaushik srinivas
Priority: Critical
Flume kafka source has been used with spillable memory channel to dump messages
into kite dataset.
Every message from a kafka topic is around 180 bytes.
And messages/sec is approx 80k/sec.
So, flume spill-able memory channel has been configured with,
memory capacity : 5000000
transaction capacity : 50000
batch size : 50000
byteCapacityBufferPercentage : 30
Upon addition of parameter byteCapacity with value 900000000 (memoryCapacity *
size of each message/event i.e 5000000*180),
We see the kafka bytes out/sec due to flume kafka source is very less compared
to kafka bytes in/sec.
If byteCapacity is removed from flume config, the bytesOut/sec seems to
increase and equal to bytesIn/sec.
What is the purpose of having byteCapacity and what should be the right way to
calculate the same ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
