Tom Bentley created KAFKA-10846:
-----------------------------------
Summary: FileStreamSourceTask buffer can grow without bound
Key: KAFKA-10846
URL: https://issues.apache.org/jira/browse/KAFKA-10846
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Tom Bentley
Assignee: Tom Bentley
When reading a large file the buffer used by {{FileStreamSourceTask}} can grow
without bound. Even in the unit test
org.apache.kafka.connect.file.FileStreamSourceTaskTest#testBatchSize the buffer
grows from 1,024 to 524,288 bytes just reading 10,000 copies of a line of <100
chars.
The problem is that the condition for growing the buffer is incorrect. The
buffer is doubled whenever some bytes were read and the used space in the
buffer == the buffer length.
The requirement to increase the buffer size should be related to whether
{{extractLine()}} actually managed to read any lines. It's only when no
complete lines were read since the last call to {{read()}} that we need to
increase the buffer size (to cope with the large line).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
