Prabhat Verma created KAFKA-6831:
------------------------------------
Summary: FileStreamSink is very slow
Key: KAFKA-6831
URL: https://issues.apache.org/jira/browse/KAFKA-6831
Project: Kafka
Issue Type: Test
Components: consumer
Affects Versions: 1.1.0
Reporter: Prabhat Verma
Fix For: 1.1.0
Hi Team,
I am very new in kafka. My project requirement is fetch data from source
location and place it in other other location (consumer location). I am using
FileStreamSink class to perform above action.
I am using Linux machine having memory of 32 GB.
When i start FIleStreamSink , It is syncing to consumer location very very
slowly. Not sure why it is taking 2000 message at a time and then sync it.
After that it wait for few second then sync again. This waiting time increases
per run .
I am processing 600K message but it took 1 hrs to process only 60K message.
Below are my config details :
connect-file-sink.property
Name = local-file
Connector.class = FileStreamSource
task.max=20
file=/d/d1/kafka/destination/outfile.txt
topic=abc_partion_20
connect-file-source.property
Name = local-file
Connector.class = FileStreamSource
task.max=20
file=/d/d1/kafka/source/infile.txt
topic=abc_partion_20
Can you please help ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
