Re: mirrormaker tool in 0.82beta
No, I missed that. thanks, svante 2015-01-07 6:44 GMT+01:00 Jun Rao j...@confluent.io: Did you set offsets.storage to kafka in the consumer of mirror maker? Thanks, Jun On Mon, Jan 5, 2015 at 3:49 PM, svante karlsson s...@csi.se wrote: I'm using 0.82beta and I'm trying to push data with the mirrormaker tool from several remote sites to two datacenters. I'm testing this from a node containing zk, broker and mirrormaker and the data is pushed to a normal cluster. 3 zk and 4 brokers with replication. While the configuration seems straight forward things are a bit shaky. First, if something goes wrong on either consumer or producer side it stops transmitting data without dying which makes it harder to run under a supervisor (upstart in my case) I start pushing data to site 02 and mirrormaker pushes it to my datacenter - so far all good... But, when I run. (on my remote site) kafka-consumer-offset-checker.sh --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106 Could not fetch offset for [saka.test.ext_datastream,7] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,4] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,2] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,5] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,6] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,3] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,1] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,0] due to kafka.common.NotCoordinatorForConsumerException. Group Topic Pid Offset logSize Lag Owner mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0 unknown 9310560 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1 unknown 9313497 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2 unknown 9323623 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3 unknown 9334005 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4 unknown 9324176 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5 unknown 9336504 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6 unknown 9316139 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7 unknown 9318770 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 Since I can push data between my sites something is working but what can be the case for NotCoordinatorForConsumerException on a single node cluster? Finally the __consumer_offsets topic should probably be masked out from mirroring without using whitelists or blacklists thanks /svante
Re: mirrormaker tool in 0.82beta
Did you set offsets.storage to kafka in the consumer of mirror maker? Thanks, Jun On Mon, Jan 5, 2015 at 3:49 PM, svante karlsson s...@csi.se wrote: I'm using 0.82beta and I'm trying to push data with the mirrormaker tool from several remote sites to two datacenters. I'm testing this from a node containing zk, broker and mirrormaker and the data is pushed to a normal cluster. 3 zk and 4 brokers with replication. While the configuration seems straight forward things are a bit shaky. First, if something goes wrong on either consumer or producer side it stops transmitting data without dying which makes it harder to run under a supervisor (upstart in my case) I start pushing data to site 02 and mirrormaker pushes it to my datacenter - so far all good... But, when I run. (on my remote site) kafka-consumer-offset-checker.sh --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106 Could not fetch offset for [saka.test.ext_datastream,7] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,4] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,2] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,5] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,6] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,3] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,1] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,0] due to kafka.common.NotCoordinatorForConsumerException. Group Topic Pid Offset logSize Lag Owner mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0 unknown 9310560 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1 unknown 9313497 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2 unknown 9323623 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3 unknown 9334005 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4 unknown 9324176 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5 unknown 9336504 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6 unknown 9316139 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7 unknown 9318770 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 Since I can push data between my sites something is working but what can be the case for NotCoordinatorForConsumerException on a single node cluster? Finally the __consumer_offsets topic should probably be masked out from mirroring without using whitelists or blacklists thanks /svante
mirrormaker tool in 0.82beta
I'm using 0.82beta and I'm trying to push data with the mirrormaker tool from several remote sites to two datacenters. I'm testing this from a node containing zk, broker and mirrormaker and the data is pushed to a normal cluster. 3 zk and 4 brokers with replication. While the configuration seems straight forward things are a bit shaky. First, if something goes wrong on either consumer or producer side it stops transmitting data without dying which makes it harder to run under a supervisor (upstart in my case) I start pushing data to site 02 and mirrormaker pushes it to my datacenter - so far all good... But, when I run. (on my remote site) kafka-consumer-offset-checker.sh --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106 Could not fetch offset for [saka.test.ext_datastream,7] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,4] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,2] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,5] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,6] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,3] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,1] due to kafka.common.NotCoordinatorForConsumerException. Could not fetch offset for [saka.test.ext_datastream,0] due to kafka.common.NotCoordinatorForConsumerException. Group Topic Pid Offset logSize Lag Owner mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0 unknown 9310560 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1 unknown 9313497 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2 unknown 9323623 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3 unknown 9334005 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4 unknown 9324176 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5 unknown 9336504 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6 unknown 9316139 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7 unknown 9318770 unknown mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0 Since I can push data between my sites something is working but what can be the case for NotCoordinatorForConsumerException on a single node cluster? Finally the __consumer_offsets topic should probably be masked out from mirroring without using whitelists or blacklists thanks /svante
