The kafka-connect-hdfs just hangs if the "offset" that it expects is no
longer present (this happens when the message get deleted because of
retention time)

The process in this case does not write any output and the messages get
ignored.

Is this by design ?

The relevant code is

TopicPartitionWriter.java

if (offset == -1) {
      offset = record.kafkaOffset();
    } else if (record.kafkaOffset() != expectedOffset) {
      // Currently it's possible to see stale data with the wrong offset
after a rebalance when you
      // rewind, which we do since we manage our own offsets. See
KAFKA-2894.
      if (!sawInvalidOffset) {
        log.info(
            "Ignoring stale out-of-order record in {}-{}. Has offset {}
instead of expected offset {}",
            record.topic(), record.kafkaPartition(), record.kafkaOffset(),
expectedOffset);
      }
      sawInvalidOffset = true;
      return;
    }

In the "else if" we should not ignore the message if the
record.kafkaOffset() is greater than expectedOffset. Any thoughts ?

Thanks,
Prabhu

Reply via email to