I have an application producing Avro-encoded keyed messages (Martin Kleppmann's new Bottled Water project).
It encodes a delete as a keyed message with an id as a key and a null payload. I have log compaction turned on. The Avro console consumer correctly displays this as "null" in my terminal, but when I try to consume it using the high-level consumer in Java, the message is never consumed. Subsequent non-null messages that were produced after that null also aren't consumed. Do I need to do something in order to have the iterator's hasNext() method (my code is pretty much exactly what appears in https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example ) not treat a null value as not being a valid message to consume or something? Or am I misunderstanding what's going on and need to do something different? Thanks!