What is your poll time for consumers poll() method?
On Thu, 30 Aug 2018, 16:23 Cristian Petroaca, <cpetro...@fitbit.com.invalid> wrote: > Ok, so I mixed things up a little. > I started with the kafka Server being configured to auto create topics. > That gave the error. > But turning the auto create off and creating the topic with AdminUtils > does not show the error and the consumer actually polls for messages. > I did not modify the “default.replication.factor” for auto created topics > and that has as default 1. So I’m not sure why I would see the error in the > first place? > > Even though I don’t see the error anymore and my consumer polls for > messages, it does not receive any messages. I am waiting a reasonable > amount of time (1min) after the producer created the messages. > An independent console consumer connected to the same borker and topic > does receive them. > My consumer config does not seem exotic as in to create such a situation. > Any reason for not receiving messages? > > Thanks > > On 30/08/2018, 11:25, "Cristian Petroaca" <cpetro...@fitbit.com.INVALID> > wrote: > > Yes. > In my programmatic env I first create it with: > AdminUtils.createTopic(zkUtils, topic, 1, 1, new Properties(), > RackAwareMode.Enforced$.MODULE$); > So partitions = 1 and replication = 1. > > The same for the remote broker, I created the topic –partitions 1 > –replication-factor 1 > > Are there any other reasons for that error? > > On 29/08/2018, 18:06, "M. Manna" <manme...@gmail.com> wrote: > > Does the topic exist in both your programmatic broker and remote > broker? > > > Also, are the topic settings same for partitions and replication > factor? > GROUP_COORDINATOR_NOT_AVAILABLE is enforced as of 0.11.x if the > auto-created topic partition/replication-factor setup doesn't match > with server's config. So you might want to check all these. > > Regards, > > On Wed, 29 Aug 2018 at 15:55, Cristian Petroaca > <cpetro...@fitbit.com.invalid> wrote: > > > Tried it, same problem with 9092. > > By the way, the same consumer works with a remote 1.0.1 Kafka > broker with > > the same config. > > There doesn’t seem to be any networking issues with the embedded > one since > > the consumer successfully sends Find Coordinator messages to it > and the > > broker responds with Coordinator not found. > > > > > > On 29/08/2018, 17:46, "M. Manna" <manme...@gmail.com> wrote: > > > > So have you tried binding it to 9092 rather than randomising > it, and > > see if > > that makes any difference? > > > > On Wed, 29 Aug 2018 at 15:41, Cristian Petroaca > > <cpetro...@fitbit.com.invalid> wrote: > > > > > Port = 0 means Kafka will start listening on a random port > which I > > need. > > > I tried it with 5000 but I get the same result. > > > > > > > > > On 29/08/2018, 16:46, "M. Manna" <manme...@gmail.com> > wrote: > > > > > > Can you extend the auto.commit.interval.ms to 5000 ? > and retry? > > Also, > > > why > > > is your port set to 0? > > > > > > Regards, > > > > > > On Wed, 29 Aug 2018 at 14:25, Cristian Petroaca > > > <cpetro...@fitbit.com.invalid> wrote: > > > > > > > Hi, > > > > > > > > I’m using the Kafka lib with version 2.11_1.0.1. > > > > I use the KafkaServer.scala class to > programmatically create a > > Kafka > > > > instance and connect it to a programmatically > created Zookeeper > > > instance. > > > > It has the following properties: > > > > host.name", "127.0.0.1" > > > > "port", "0" > > > > "zookeeper.connect", "127.0.0.1:" + zooKeeperPort > > > > "broker.id", "1" > > > > auto.create.topics.enable", "true" > > > > "delete.topic.enable", "true" > > > > > > > > I then create a new Kafka Consumer with the following > > properties: > > > > bootstrap.servers", “127.0.0.1” + kafkaPort > > > > "auto.commit.interval.ms", "10" > > > > “client_id”, “xxxx” > > > > “enable.auto.commit”, “true” > > > > “auto.commit.interval.ms”, “10” > > > > > > > > My problem is that after I subscribe the consumer to > a custom > > topic, > > > the > > > > consumer just blocks in the .poll() method and I see > a lot of > > > messages like: > > > > “Group coordinator lookup failed: The coordinator is > not > > available.” > > > > > > > > I read on another forum that a possible problem is > that the > > > > _consumer_offsets topic doesn’t exist but that’s not > the case > > for me. > > > > > > > > Can you suggest a possible root cause? > > > > > > > > Thanks, > > > > Cristian > > > > > > > > > > > > > > > > > > > > > > > >
