hey, i am just getting started with kafka + spark structured streaming. so this is probably a pretty dumb mistake.
i wrote a little program in spark to read messages from a kafka topic and display them in the console, using the kafka source and console sink. i run it it in spark local mode. i hooked it up to a test topic that i send messages to using the kafka console producer, and everything works great. i type a message in the console producer, and it pops up in my spark program. very neat! next i point it to another topic instead on which a kafka-connect program is writing lots of irc messages. i can see kafka connect to the topic successfully, the partitions are discovered etc., and then... nothing. it just keeps stuck at offsets 0 for all partitions. at the same time in another terminal i can see messages coming in just fine using the kafka console consumer. i dont get it. why doesnt kafka want to consume from this topic in spark structured streaming? thanks! koert
