Hi Chris, Thanks for the pointers, very useful. I got my tests to pass but seem to have broken the TopicMetadataCache in the process; I assume somewhere internally it's using the client id (not group id) as part of an equality test. But I'll dig into it tomorrow.
Thanks, Garry -----Original Message----- From: Chris Riccomini [mailto:[email protected]] Sent: 19 November 2013 01:07 To: [email protected] Subject: Re: Problem testing creation of ConsumerConfig object (re SAMZA-77) Hey Gary, Wow, great! :) Kafka, does indeed have some requirements on what config parameters it needs, and it does verify these at instantiation time. The parameters you'll need to set are: systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactor y systems.kafka.consumer.zookeeper.connect=localhost:2181/ systems.kafka.producer.metadata.broker.list=localhost:9092 Note that, in this case, the system is called "kafka". In your example, you'd use the "systems.appname.*" prefix (or change your test to call getKafkaSystemConsumerConfig("kafka") ). Kafka does come with a way to do this in their TestUtils class: https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/u tils/TestUtils.scala?source=c The method you're probably looking for is: createConsumerProperties And: getProducerConfig What I'd recommend is just creating a java.util.Properties instance, and adding all appropriate configs to it (either with putAll, or put). Then I'd use the MapConfig class to convert the Properties to a Config object: val config = new MapConfig(myProps); This is a bit easier than going the factory route that you're on right now, and it can all be done inside your unit test, instead of having extra properties files. Cheers, Chris On 11/18/13 4:53 PM, "Garry Turkington" <[email protected]> wrote: >Hi, > >I thought SAMZA-77 looked like fun so made the changes required to add >the stringified UUID to the client and group IDs. All nice and easy. >Then I tried adding some test cases and think I'm managing to make this >more difficult than it needs to be. :) > >In my test case I have the following, which I believe is the right >chain of calls to create a KafkaConfig object (I created a simple >config file in the resources directory): > > val factory = new PropertiesConfigFactory() > val config = >factory.getConfig(URI.create("file://%s/src/test/resources/test.propert >ies <file:///\\%25s\src\test\resources\test.properties>" format new >File >(".").getCanonicalPath)) >val kafkaConfig = new KafkaConfig(config) > >All well and good but then I try and call the appropriate method on the >KafkaConfig object (with some uncertainty of what the argument needs >map >to): > >val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname") > >This causes an exception to be thrown, the highlights being: > >java.lang.IllegalArgumentException: requirement failed: Missing >required property 'zookeeper.connect' > at scala.Predef$.require(Predef.scala:145) > at >kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175) > at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775) > at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73) > at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77) > at >org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaC >onf >ig.scala:91) > >This suggests that to create a ConsumerConfig requires the whole >ZK/Kafka stack to be instantiated - is this the case and if so pointers >to the minimum config set required to get it working? I tried to piece >something together from the other test cases but couldn't find the >right magic. > >Or is there an easier way to do all this? :) > >Thanks >Garry ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.3426 / Virus Database: 3629/6841 - Release Date: 11/16/13
