RE: Looks like consumer fetchers get stopped we are not getting any data
Currently its set to default,whats the timeout should we increase to.
From: Guozhang Wang [wangg...@gmail.com]
Sent: Saturday, January 11, 2014 3:53 PM
To: users@kafka.apache.org
Subject: Re: Looks like consumer fetchers get stopped we are not getting any
data
>From the logs it seems the consumer's ZK registry log has lost, while
KAFKA-693 is mainly due to server side issue. Could you check if there is a
session timeout from the consumer on ZK log.
Guozhang
On Sat, Jan 11, 2014 at 2:33 PM, Seshadri, Balaji
wrote:
> We found the below bug in Kafka JIRA.
>
> https://issues.apache.org/jira/browse/KAFKA-693
>
> Could you guys please let us know if this bug is causing the fetchers
> getting stooped because of rebalance failure,we are using 0.8-beta and this
> fix seems to be in 0.8 version.
>
> The issue happens only when we run it in tomcat,when we use JUnits with
> same group.id to test we receive the messages.It would be great if you
> could help us confirm if it is a bug.
>
> Also we would like to know if we need to upgrade the server/client to 0.8
> or just server side code.
>
>
>
> From: Rob Withers [robert.w.with...@gmail.com]
> Sent: Friday, January 10, 2014 10:17 PM
> To: users@kafka.apache.org
> Subject: Re: Looks like consumer fetchers get stopped we are not getting
> any data
>
> That was an interesting section too. Which GC settings would you suggest?
>
> Thank you,
> - charlie
>
> > On Jan 10, 2014, at 10:11 PM, Jun Rao wrote:
> >
> > Have you looked at our FAQ, especially
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > ?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Jan 10, 2014 at 2:25 PM, Seshadri, Balaji
> > wrote:
> >
> >> Any clue would be helpful.
> >>
> >> -Original Message-
> >> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> >> Sent: Friday, January 10, 2014 12:46 PM
> >> To: users@kafka.apache.org
> >> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> >> any data
> >>
> >> Yes rebalance begins and exceptions occurs.
> >>
> >>
> >> {2014-01-10 00:58:11,293} INFO
> >>
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> >> (?:?) - [account-i
> >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Cleared
> the
> >> data chunks in all the consumer message iterators
> >> {2014-01-10 00:58:11,293} INFO
> >>
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> >> (?:?) - [account-i
> >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> Committing
> >> all offsets after clearing the fetcher queues
> >> {2014-01-10 00:58:11,298} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:759) - Got ping
> >> response for sessionid: 0x1437b2879870005 af ter 0ms
> >> {2014-01-10 00:58:11,313} INFO
> >>
> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
> >> (?:?) -
> >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
> >> begin rebalancing consumer
> >> account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738 try
> #1
> >> {2014-01-10 00:58:11,314} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 627,8 replyHeader:: 627,51539619966,0
> request::
> >> '/brokers/ids,F response:: v{'1}
> >> {2014-01-10 00:58:11,315} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 628,4 replyHeader:: 628,51539619966,0
> request::
> >> '/brokers/ids/1,F response::
> >>
> #7b2022686f7374223a22746d312d6b61666b6162726f6b6572313031222c20226a6d785f706f7274223a393939392c2022706f7274223a393039322c202276657273696f6e223a31207d,s{47244644685,47244644685,1388537628753,1388537628753,0,0,0,163056791896588316,74,0,47244644685}
> >> {2014-01-10 00:58:11,316} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 629,4 replyHeader:: 629,51539619966,-101
> >> request::
> >>
> '/consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738,F
> >> response::
> >> {2014-01-10 00:58:11,316} INFO
> >>
> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
> >> (?:?) -
> >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
> >> exception during rebalance
> >> org.I0Itec.zkclient.exception.ZkNoNodeException:
> >> org
restoring broker node / zookeeper question
I have two questions: 1. We plan to initially run Kafka without partitioning or replication, simply having two single node Kafka (broker) servers in two geographically diverse locations. Using periodic backups/snapshots across locations for disaster recovery. Our producer / consumer applications will process to/from each location. For such configurations, is it possible to run the Kafka broker without Zookeeper? If not, any plans for such a simplified configuration? We'd like to remove the extra app from the stack (Zookeeper) since it's not being utilized. 2. If we need to restore a Kafka node (simplified design above), I was hoping Kafka and Zookeeper could bootstrap itself as long as we can restore a copy of the /tmp/kafka-logs data. This dir contains the topics, log data, and high watermark. Unfortunately, it seems without the Zookeeper data, we cannot restore the node. Is there a way to restore a Kafka node if the Zookeeper data cannot be restored? What are the specific requirements to restore a Kafka broker from backups, if the /tmp/zookeeper must also be restored, are there any inconsistent state issues (i.e. backup of /tmp/kafka-logs and /tmp/zookeeper are not synchronized)? Thank you!
RE: Looks like consumer fetchers get stopped we are not getting any data
We have attached more logs where the fetchers are stopped.
https://issues.apache.org/jira/browse/KAFKA-693 issue mentions about this
issue, and do we need to upgrade to 0.8.0 release for this fix?
Regards,
RJ
-Original Message-
From: Seshadri, Balaji
Sent: Saturday, January 11, 2014 3:34 PM
To: users@kafka.apache.org
Cc: Gulia, Vikram; Nanjegowda, Mithunraj; Behera, Ajit; Withers, Robert;
ShankenpurMayanna, Diwakar
Subject: RE: Looks like consumer fetchers get stopped we are not getting any
data
We found the below bug in Kafka JIRA.
https://issues.apache.org/jira/browse/KAFKA-693
Could you guys please let us know if this bug is causing the fetchers getting
stooped because of rebalance failure,we are using 0.8-beta and this fix seems
to be in 0.8 version.
The issue happens only when we run it in tomcat,when we use JUnits with same
group.id to test we receive the messages.It would be great if you could help us
confirm if it is a bug.
Also we would like to know if we need to upgrade the server/client to 0.8 or
just server side code.
From: Rob Withers [robert.w.with...@gmail.com]
Sent: Friday, January 10, 2014 10:17 PM
To: users@kafka.apache.org
Subject: Re: Looks like consumer fetchers get stopped we are not getting any
data
That was an interesting section too. Which GC settings would you suggest?
Thank you,
- charlie
> On Jan 10, 2014, at 10:11 PM, Jun Rao wrote:
>
> Have you looked at our FAQ, especially
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> ?
>
> Thanks,
>
> Jun
>
>
> On Fri, Jan 10, 2014 at 2:25 PM, Seshadri, Balaji
> wrote:
>
>> Any clue would be helpful.
>>
>> -Original Message-
>> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
>> Sent: Friday, January 10, 2014 12:46 PM
>> To: users@kafka.apache.org
>> Subject: RE: Looks like consumer fetchers get stopped we are not getting
>> any data
>>
>> Yes rebalance begins and exceptions occurs.
>>
>>
>> {2014-01-10 00:58:11,293} INFO
>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
>> (?:?) - [account-i
>> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Cleared the
>> data chunks in all the consumer message iterators
>> {2014-01-10 00:58:11,293} INFO
>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
>> (?:?) - [account-i
>> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Committing
>> all offsets after clearing the fetcher queues
>> {2014-01-10 00:58:11,298} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:759) - Got ping
>> response for sessionid: 0x1437b2879870005 af ter 0ms
>> {2014-01-10 00:58:11,313} INFO
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
>> (?:?) -
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
>> begin rebalancing consumer
>> account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738 try #1
>> {2014-01-10 00:58:11,314} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 627,8 replyHeader:: 627,51539619966,0 request::
>> '/brokers/ids,F response:: v{'1}
>> {2014-01-10 00:58:11,315} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 628,4 replyHeader:: 628,51539619966,0 request::
>> '/brokers/ids/1,F response::
>> #7b2022686f7374223a22746d312d6b61666b6162726f6b6572313031222c20226a6d785f706f7274223a393939392c2022706f7274223a393039322c202276657273696f6e223a31207d,s{47244644685,47244644685,1388537628753,1388537628753,0,0,0,163056791896588316,74,0,47244644685}
>> {2014-01-10 00:58:11,316} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 629,4 replyHeader:: 629,51539619966,-101
>> request::
>> '/consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738,F
>> response::
>> {2014-01-10 00:58:11,316} INFO
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
>> (?:?) -
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
>> exception during rebalance
>> org.I0Itec.zkclient.exception.ZkNoNodeException:
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
>> NoNode for
>> /consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738
>>at
>> org.I0Itec.zkclient.exception.ZkException
Re: Looks like consumer fetchers get stopped we are not getting any data
>From the logs it seems the consumer's ZK registry log has lost, while
KAFKA-693 is mainly due to server side issue. Could you check if there is a
session timeout from the consumer on ZK log.
Guozhang
On Sat, Jan 11, 2014 at 2:33 PM, Seshadri, Balaji
wrote:
> We found the below bug in Kafka JIRA.
>
> https://issues.apache.org/jira/browse/KAFKA-693
>
> Could you guys please let us know if this bug is causing the fetchers
> getting stooped because of rebalance failure,we are using 0.8-beta and this
> fix seems to be in 0.8 version.
>
> The issue happens only when we run it in tomcat,when we use JUnits with
> same group.id to test we receive the messages.It would be great if you
> could help us confirm if it is a bug.
>
> Also we would like to know if we need to upgrade the server/client to 0.8
> or just server side code.
>
>
>
> From: Rob Withers [robert.w.with...@gmail.com]
> Sent: Friday, January 10, 2014 10:17 PM
> To: users@kafka.apache.org
> Subject: Re: Looks like consumer fetchers get stopped we are not getting
> any data
>
> That was an interesting section too. Which GC settings would you suggest?
>
> Thank you,
> - charlie
>
> > On Jan 10, 2014, at 10:11 PM, Jun Rao wrote:
> >
> > Have you looked at our FAQ, especially
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > ?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Jan 10, 2014 at 2:25 PM, Seshadri, Balaji
> > wrote:
> >
> >> Any clue would be helpful.
> >>
> >> -Original Message-
> >> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
> >> Sent: Friday, January 10, 2014 12:46 PM
> >> To: users@kafka.apache.org
> >> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> >> any data
> >>
> >> Yes rebalance begins and exceptions occurs.
> >>
> >>
> >> {2014-01-10 00:58:11,293} INFO
> >>
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> >> (?:?) - [account-i
> >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Cleared
> the
> >> data chunks in all the consumer message iterators
> >> {2014-01-10 00:58:11,293} INFO
> >>
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> >> (?:?) - [account-i
> >> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> Committing
> >> all offsets after clearing the fetcher queues
> >> {2014-01-10 00:58:11,298} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:759) - Got ping
> >> response for sessionid: 0x1437b2879870005 af ter 0ms
> >> {2014-01-10 00:58:11,313} INFO
> >>
> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
> >> (?:?) -
> >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
> >> begin rebalancing consumer
> >> account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738 try
> #1
> >> {2014-01-10 00:58:11,314} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 627,8 replyHeader:: 627,51539619966,0
> request::
> >> '/brokers/ids,F response:: v{'1}
> >> {2014-01-10 00:58:11,315} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 628,4 replyHeader:: 628,51539619966,0
> request::
> >> '/brokers/ids/1,F response::
> >>
> #7b2022686f7374223a22746d312d6b61666b6162726f6b6572313031222c20226a6d785f706f7274223a393939392c2022706f7274223a393039322c202276657273696f6e223a31207d,s{47244644685,47244644685,1388537628753,1388537628753,0,0,0,163056791896588316,74,0,47244644685}
> >> {2014-01-10 00:58:11,316} DEBUG [catalina-exec-12-SendThread(
> >> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
> >> reply sessionid:0x1437b2879870005, packet:: clientPath:null
> serverPath:null
> >> finished:false header:: 629,4 replyHeader:: 629,51539619966,-101
> >> request::
> >>
> '/consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738,F
> >> response::
> >> {2014-01-10 00:58:11,316} INFO
> >>
> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
> >> (?:?) -
> >> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
> >> exception during rebalance
> >> org.I0Itec.zkclient.exception.ZkNoNodeException:
> >> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> >> NoNode for
> >>
> /consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738
> >>at
> >> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> >>
RE: Looks like consumer fetchers get stopped we are not getting any data
We found the below bug in Kafka JIRA.
https://issues.apache.org/jira/browse/KAFKA-693
Could you guys please let us know if this bug is causing the fetchers getting
stooped because of rebalance failure,we are using 0.8-beta and this fix seems
to be in 0.8 version.
The issue happens only when we run it in tomcat,when we use JUnits with same
group.id to test we receive the messages.It would be great if you could help us
confirm if it is a bug.
Also we would like to know if we need to upgrade the server/client to 0.8 or
just server side code.
From: Rob Withers [robert.w.with...@gmail.com]
Sent: Friday, January 10, 2014 10:17 PM
To: users@kafka.apache.org
Subject: Re: Looks like consumer fetchers get stopped we are not getting any
data
That was an interesting section too. Which GC settings would you suggest?
Thank you,
- charlie
> On Jan 10, 2014, at 10:11 PM, Jun Rao wrote:
>
> Have you looked at our FAQ, especially
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> ?
>
> Thanks,
>
> Jun
>
>
> On Fri, Jan 10, 2014 at 2:25 PM, Seshadri, Balaji
> wrote:
>
>> Any clue would be helpful.
>>
>> -Original Message-
>> From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
>> Sent: Friday, January 10, 2014 12:46 PM
>> To: users@kafka.apache.org
>> Subject: RE: Looks like consumer fetchers get stopped we are not getting
>> any data
>>
>> Yes rebalance begins and exceptions occurs.
>>
>>
>> {2014-01-10 00:58:11,293} INFO
>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
>> (?:?) - [account-i
>> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Cleared the
>> data chunks in all the consumer message iterators
>> {2014-01-10 00:58:11,293} INFO
>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
>> (?:?) - [account-i
>> nfo-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], Committing
>> all offsets after clearing the fetcher queues
>> {2014-01-10 00:58:11,298} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:759) - Got ping
>> response for sessionid: 0x1437b2879870005 af ter 0ms
>> {2014-01-10 00:58:11,313} INFO
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
>> (?:?) -
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
>> begin rebalancing consumer
>> account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738 try #1
>> {2014-01-10 00:58:11,314} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 627,8 replyHeader:: 627,51539619966,0 request::
>> '/brokers/ids,F response:: v{'1}
>> {2014-01-10 00:58:11,315} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 628,4 replyHeader:: 628,51539619966,0 request::
>> '/brokers/ids/1,F response::
>> #7b2022686f7374223a22746d312d6b61666b6162726f6b6572313031222c20226a6d785f706f7274223a393939392c2022706f7274223a393039322c202276657273696f6e223a31207d,s{47244644685,47244644685,1388537628753,1388537628753,0,0,0,163056791896588316,74,0,47244644685}
>> {2014-01-10 00:58:11,316} DEBUG [catalina-exec-12-SendThread(
>> tvip-m1-mw-zookeeper.dish.com:2181)] (ClientCnxn.java:839) - Reading
>> reply sessionid:0x1437b2879870005, packet:: clientPath:null serverPath:null
>> finished:false header:: 629,4 replyHeader:: 629,51539619966,-101
>> request::
>> '/consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738,F
>> response::
>> {2014-01-10 00:58:11,316} INFO
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738_watcher_executor]
>> (?:?) -
>> [account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738],
>> exception during rebalance
>> org.I0Itec.zkclient.exception.ZkNoNodeException:
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
>> NoNode for
>> /consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738
>>at
>> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>>at
>> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
>>at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
>>at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
>>at kafka.utils.ZkUtils$.readData(Unknown Source)
>>at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
>>at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebala
Re: custom kafka consumer - strangeness
I'm also seeing the following. I consume the data in the queue. Then after 10 seconds send another fetch request (with the incremented offset), and never receives a response from the broker, my code eventually times out (after 30seconds). The broker writes Expiring fetch request Name: FetchRequest; Version: 0; CorrelationId: 1389443537; ClientId: 1; ReplicaId: -1; MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [ping,0] -> PartitionFetchInfo(187,1048576). This corresponds with the timed out fetch request. On Sat, Jan 11, 2014 at 12:19 PM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > Hi, > > > No the offsets are not the same. I've printed out the values to see this, > and its not the case. > > > > On Fri, Jan 10, 2014 at 5:02 PM, Jun Rao wrote: > >> Are the offset used in the 2 fetch requests the same? If so, you will get >> the same messages twice. You consumer is responsible for advancing the >> offsets after consumption. >> >> Thanks, >> >> Jun >> >> >> On Thu, Jan 9, 2014 at 1:00 PM, Gerrit Jansen van Vuuren < >> gerrit...@gmail.com> wrote: >> >> > Hi, >> > >> > I'm writing a custom consumer for kafka 0.8. >> > Everything works except for the following: >> > >> > a. connect, send fetch, read all results >> > b. send fetch >> > c. send fetch >> > d. send fetch >> > e. via the console publisher, publish 2 messages >> > f. send fetch :corr-id 1 >> > g. read 2 messages published :offsets [10 11] :corr-id 1 >> > h. send fetch :corr-id 2 >> > i. read 2 messages published :offsets [10 11] :corr-id 2 >> > j. send fetch ... >> > >> > The problem is I get the messages sent twice as a response to two >> separate >> > fetch requests. The correlation id is distinct so it cannot be that I >> read >> > the response twice. The offsets of the 2 messages are are the same so >> they >> > are duplicates, and its not the producer sending the messages twice. >> > >> > Note: the same connection is kept open the whole time, and I send >> > block,receive then send again, after the first 2 messages are read, the >> > offsets are incremented and the next fetch will ask kafka to give it >> > messages from the new offsets. >> > >> > any ideas of why kafka would be sending the messages again on the second >> > fetch request? >> > >> > Regards, >> > Gerrit >> > >> > >
Re: custom kafka consumer - strangeness
Hi, No the offsets are not the same. I've printed out the values to see this, and its not the case. On Fri, Jan 10, 2014 at 5:02 PM, Jun Rao wrote: > Are the offset used in the 2 fetch requests the same? If so, you will get > the same messages twice. You consumer is responsible for advancing the > offsets after consumption. > > Thanks, > > Jun > > > On Thu, Jan 9, 2014 at 1:00 PM, Gerrit Jansen van Vuuren < > gerrit...@gmail.com> wrote: > > > Hi, > > > > I'm writing a custom consumer for kafka 0.8. > > Everything works except for the following: > > > > a. connect, send fetch, read all results > > b. send fetch > > c. send fetch > > d. send fetch > > e. via the console publisher, publish 2 messages > > f. send fetch :corr-id 1 > > g. read 2 messages published :offsets [10 11] :corr-id 1 > > h. send fetch :corr-id 2 > > i. read 2 messages published :offsets [10 11] :corr-id 2 > > j. send fetch ... > > > > The problem is I get the messages sent twice as a response to two > separate > > fetch requests. The correlation id is distinct so it cannot be that I > read > > the response twice. The offsets of the 2 messages are are the same so > they > > are duplicates, and its not the producer sending the messages twice. > > > > Note: the same connection is kept open the whole time, and I send > > block,receive then send again, after the first 2 messages are read, the > > offsets are incremented and the next fetch will ask kafka to give it > > messages from the new offsets. > > > > any ideas of why kafka would be sending the messages again on the second > > fetch request? > > > > Regards, > > Gerrit > > >
Re: Kafka with Docker - producer disconnecting
Hi, you can have a look at https://github.com/wurstmeister/kafka-docker maybe that helps On 10/01/2014 19:30, Alex Artigues wrote: Hi everyone, I am attempting to run Zookeeper and Kafka in dockers. Both startup normally and Kafka connects ok. I think my containers are linked fine because I am able to list topics, and create topics. The producer however never delivers messages. It connects using the provided shell scripts, and it seems I am able to type messages, but I cannot ctl+c to stop it. in the logs there is an INFO message that the client disconnects, but no good reason why. This is in the ZK logs: 2014-01-10 17:55:39,893 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1437d4304fe, likely client has closed socket If anyone has any clues I would really appreciate it, thanks.
