I'm using KafkaIO with Kafka 0.10 and the InProcessPipelineRunner. I'm not receiving any KafkaRecords in the PCollection.
In my code, I create a KafkaIO.read() that is processed by a MapElements.via(). The lamdba in the via is never called. I've verified there is data in the topic and I've manually added data using the console producer. Still, no data. The second thing is about the JavaDoc on the read method. I think the JavaDocs should talk about the differences between this and a default KafkaConsumer. Since there isn't a group.id required to be set, I think the JavaDoc should mention the implications of this. It would say something about always starting from the latest data. Also, it should mention that offsets won't be kept and restarting the processing will not start at the last offset; it will start at the latest data again. Thanks, Jesse
