We are doing one scala consumer and one java consumer who listen on same topic
with different group id.The issue is if I start both second one fails with this
error.If we run just one its working fine.
Attaching Scala code and Java code where we call createMessageStreams.It seems
like issue with API not recognizing that its different group of consumers that
are started because createMessageStream doesnot accept group.id but should
refer already passed group.id to make sure second on is new stream and not
related even we listen on same topic.
Please let me know if it is API issue or way we handle API.
-Original Message-
From: Joe Stein [mailto:joe.st...@stealth.ly]
Sent: Friday, December 13, 2013 4:36 PM
To: users@kafka.apache.org
Subject: Re: Unable to start consumers in Tomcat
you do this at the consumer connector level not at the message stream level
so one
propsA.put("group.id", "groupA") be
val configA = new ConsumerConfig(propsA) val one = Consumer.create(configA)
and another
propsB.put("group.id", "groupB") be
val configB = new ConsumerConfig(propsB) val two = Consumer.create(configB)
so you then manage each Consumer connector created by each of those .create in
separate threads (or actors or processes or however you are doing concurrency)
and you only call the create message stream each once but each once on each of
them
makes sense?
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
/
On Fri, Dec 13, 2013 at 6:29 PM, Seshadri, Balaji
wrote:
> Cant we create message stream for same topic but different consumer
> group ?.
>
> -Original Message-
> From: Joe Stein [mailto:joe.st...@stealth.ly]
> Sent: Friday, December 13, 2013 4:23 PM
> To: users@kafka.apache.org
> Subject: Re: Unable to start consumers in Tomcat
>
> You should check your code to verify that it is only called once per
> instantiated consumer connector. Here is where the exception is
> thrown
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/con
> sumer/ZookeeperConsumerConnector.scala#L133from
>
> /***
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
> /
>
>
> On Fri, Dec 13, 2013 at 6:17 PM, Seshadri, Balaji
> wrote:
>
> > We needed HTTP interface to start our consumers using REST interface
> > for management that why we chose tomcat to run our consumers.
> >
> > We create streams only once.when we initially start consumer.
> >
> > -Original Message-
> > From: Joe Stein [mailto:joe.st...@stealth.ly]
> > Sent: Friday, December 13, 2013 4:12 PM
> > To: users@kafka.apache.org
> > Subject: Re: Unable to start consumers in Tomcat
> >
> > That error comes if you are trying to create a message/stream on a
> > consumer instantiated object more than once.
> >
> > Why are you using tomcat for the consumers? Is it to see the results
> > of messages? if so you need to isolate the Consumer in some way so
> > there is a singleton (assuming one partition or if more than one
> > partition some type of consumer pool) and the messages being
> > consumed write to a thread safe memory structure and then the UI can
> > read from that thread safe memory structure.
> >
> > /***
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop
> > /
> >
> >
> > On Fri, Dec 13, 2013 at 5:42 PM, Seshadri, Balaji
> > wrote:
> >
> > > 0.8
> > >
> > > -Original Message-
> > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
> > > Sent: Friday, December 13, 2013 3:33 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Unable to start consumers in Tomcat
> > >
> > > Which version of kafka are you using?
> > >
> > >
> > > On Fri, Dec 13, 2013 at 2:29 PM, Seshadri, Balaji
> > > wrote:
> > >
> > > > Any idea on this error guys ?.
> > > >
> > > > -Original Message-
> > > > From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> > > > Sent: Friday, December 13, 2013 9:32 AM
> > > > To: 'users@kafka.apache.org'
> > > > Subject: Unable to start consumers in Tomcat
> > > >
> > > > Hi ,
> > > >
> > > > Can you guys lets us know why are we getting this error when try
> > > > to spawn a consumer ?.
> > > >
> > > > ZookeeperConsumerConnector can create message streams at most
> > > > once
> > > >
> > > > Balaji
> > > >
> > >
> >
>
