kaushik srinivas created FLUME-3210:
---------------------------------------
Summary: Flume kafka source slows down when injestion to kafka
stops.
Key: FLUME-3210
URL: https://issues.apache.org/jira/browse/FLUME-3210
Project: Flume
Issue Type: Question
Reporter: kaushik srinivas
Attachments: ByteOutKafka.PNG, BytesInKafka.PNG
Test being done:
1. Kafka topic created with replication factor 3.
2. Flume agent created with below config,
source type = org.apache.flume.source.kafka.KafkaSource
channel type = SPILLABLEMEMORY
sink type = org.apache.flume.sink.kite.DatasetSink
3. Kafka data producers started to pump traffic to kafka topic.
4. Flume starts polling and consume records.
5. Observed kafka bytesIn/sec and bytesout/sec.
Bytesout/sec was found to be approx ~3 times of bytesin/sec when the data
injestion to
kafka is still happening from producer.
6. Kafka producer code stopped.
Observed BytesIn/sec stopped using graphana graphs.
But now the Bytesout/sec comes down to a very low value and flume continues to
consume the topic at a very low and constant rate for a very long time.
There is a huge latency observed till the flume completes the consumption of
the topic.
What could be the issue here and is there a way to identiy if the flume kafka
source is running slow or not ? No exceptions or warnings are observed in the
flume logs at anytime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
