Thanks Harsha, Looks like ZkStringSerializer was the key to creating topics via AdminUtils. That’s quite an easy mistake to make. I now noticed that the issue is already being addressed at https://issues.apache.org/jira/browse/KAFKA-1737 <https://issues.apache.org/jira/browse/KAFKA-1737>, great job!
Thanks everyone for your help! Best regards Henri > On 14.1.2015, at 4.26, Harsha <ka...@harsha.io> wrote: > > Internally producer sends a TopicMetadataRequest which creates the topic > if auto.create.topics.enable is true. > > On Tue, Jan 13, 2015, at 04:49 PM, Harsha wrote: >> >> You can import ZKStringSerializer from kafka.utils.ZkClient or write >> your own similar string serializer like this >> https://gist.github.com/harshach/7b5447c39168eb6062e0 >> >> On Tue, Jan 13, 2015, at 04:31 PM, Harsha wrote: >>> It looks like you are not adding a ZkStringSerializer to your zkClient. >>> >>> ZkClient zkClient = new ZkClient(ZK_CONN_STRING); >>> in Kafka TopicCommand uses >>> "new ZkClient(opts.options.valueOf(opts.zkConnectOpt), 30000, 30000, >>> ZKStringSerializer)" >>> >>> because of this although your topic is getting created and a zookeeper >>> path created. Zk path contents are messed up >>> get /brokers/topics/test_d5ee7dfe-cf33-4135-9bbd-c1e7d9367af2 >>> ��t${"version":1,"partitions":{"0":[0]}} >>> cZxid = 0x2ea0 >>> ctime = Tue Jan 13 16:08:19 PST 2015 >>> mZxid = 0x2ea0 >>> mtime = Tue Jan 13 16:08:19 PST 2015 >>> >>> >>> >>> On Tue, Jan 13, 2015, at 02:28 PM, Henri Pihkala wrote: >>>> Hi, >>>> >>>> Thanks for your reply, Guozhang, you are right. >>>> >>>> I eventually got it working, but it required TWO changes to what I >>>> initially tried: >>>> >>>> 1) Use OffsetRequest, not OffsetFetchRequest, like Guozhang pointed out. >>>> >>>> 2) Create topics by sending a TopicMetadataRequest and having >>>> auto.create.topics.enable=true. Initially I tried to create topics >>>> explicitly using AdminUtils.createTopic(), but this did not work for some >>>> reason! >>>> >>>> Do you think the problem I had in 2) is worth investigating or reporting >>>> further? The gist below is a test case that passes as is, but if I change >>>> the topic creation method to AdminUtils.createTopic() the test fails. Can >>>> anyone explain why? >>>> >>>> https://gist.github.com/hpihkala/f93ef3dc4c08382df69e >>>> <https://gist.github.com/hpihkala/f93ef3dc4c08382df69e> >>>> >>>> Best regards >>>> Henri >>>> >>>> >>>>> On 13.1.2015, at 19.34, Guozhang Wang <wangg...@gmail.com> wrote: >>>>> >>>>> Henri, >>>>> >>>>> FetchOffsetRequest is used for retrieving the last committed offset of a >>>>> certain consumer group; for your use case you would send OffsetRequest. >>>>> >>>>> That said the error code is misleading, it may be a bug in not indicating >>>>> the right error code. >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> On Fri, Jan 9, 2015 at 4:10 AM, Henri Pihkala <henri.pihk...@streamr.com> >>>>> wrote: >>>>> >>>>>> (1) Create topic >>>>>> (2) Send a FetchOffsetRequest for a partition in the new topic >>>>>> >>>>>> Results in error code 3, or UnknownTopicOrPartition. >>>>>> >>>>>> How can it be unknown, since I just created it? Shouldn’t it return 0, as >>>>>> that’s the offset of the next (first) message? >>>>>> >>>>>> Is this a bug or a feature? Tried with both 0.8.1.1 and 0.8.2-beta >>>>>> (single-node cluster). >>>>>> >>>>>> Best regards >>>>>> Henri >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- Guozhang >>>>
