Hi!
I sometimes get the following error in one of my streaming pipelines that
use KafkaIO as sink:
java.io.IOException: KafkaWriter : failed to send 1 records (since last report)
org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:120)
org.apache.beam.sdk.io.kafka.KafkaWriter.finishBundle(KafkaWriter.java:74)
Caused by: org.apache.kafka.common.errors.KafkaStorageException: Disk
error when trying to access log file on the disk.
My Kafka setup is not super beefy and as I understand it this seems to
happen when it is under heavy load from my Dataflow job.
What I am wondering is essentially if this means that I am loosing data or
if this will be retried by the sink? Also if this means losing the record
what is the best way to configure the KafkaIO sink to be less aggressive?
I am still Beam 2.8 in this pipeline.
Regards,
Vilhelm von Ehrenheim
