The kafka-connect-hdfs just hangs if the "offset" that it expects is no longer present (this happens when the message get deleted because of retention time)
The process in this case does not write any output and the messages get ignored. Is this by design ? The relevant code is TopicPartitionWriter.java if (offset == -1) { offset = record.kafkaOffset(); } else if (record.kafkaOffset() != expectedOffset) { // Currently it's possible to see stale data with the wrong offset after a rebalance when you // rewind, which we do since we manage our own offsets. See KAFKA-2894. if (!sawInvalidOffset) { log.info( "Ignoring stale out-of-order record in {}-{}. Has offset {} instead of expected offset {}", record.topic(), record.kafkaPartition(), record.kafkaOffset(), expectedOffset); } sawInvalidOffset = true; return; } In the "else if" we should not ignore the message if the record.kafkaOffset() is greater than expectedOffset. Any thoughts ? Thanks, Prabhu