I'm firing up a KafkaServer (using some EmbeddedKafkaBroker code that I found on Github) so that I can run an end-to-end test ingesting data through a kafka topic with consumers in Spark Streaming pushing to Accumulo.
Thus far, my code is doing this: 1) Creating a MiniAccumuloCluster and KafkaServer (using the zookeepers from the mini Accumulo cluster) 2) Creating a topic using AdminUtil 3) Starting up a Spark streaming context using a Kafka stream that puts all data into Accumulo 4) Creating a producer and sending a message to the Kafka topic. Looking @ the topic metadata in zookeeper after the topic is created, let's say "testtopic", I never see the metadata for a leader in /brokers/topics/testtopic/partitions show up. If I understand correctly, creating a topic does this: 1) Adds a persistent node into Zookeeper with some json data to denote the topic's name as well as the partitions and the list of each broker id for each partition. 2) I am still in the process of digging into this part but I think the first item in the list of replicas for each partition is used to define the initial leader and the leader is notified via a watcher and told to create an ephemeral node so that it can know when that node goes down to assign another. If I'm correct about #2, it seems like that watcher is never being invoked. Any attempt to produce to the topic just returns an error back to the producer that says there was no leader selected. Anything advice would be much appreciated. I really would like to get our stack tested fully through automated testing and Kafka is the last piece we need to assemble.
