Hi all, I have started using KafkaIO to read a data stream and have the following questions. Appreciate it if you could provide a few clarifications on the following.
1. Does KafkaIO ignore the offset stored in the broker and uses the offset stored during checkpointing when consuming messages? 2. How many threads will be used by the Kafka consumer? 3. If the consumer polls a set of messages A and then later B while A is still being processed, is there a possibility of set B finishing before A? Does parallelism control this? 4. In the above scenario if B is committed back to the broker and somehow A failed, upon a restart is there any way we can consume A again without losing data? Thank you.
