migrating log data to new locations

2014-10-07 Thread Javier Alba
Hi, I have a Kafka 0.8.1.1 cluster consisting in 4 servers with several topics on it. The cluster was initially configured to store kafka log data in a single directory on each server (log.dirs = /tmp/kafka-logs) Now, I have assigned 3 new disks to each server and I would like to use them to

No process listening on advertised.port

2014-10-07 Thread Biju N
Hello There, I have the following in my server.properties on a two node kafka test cluster …. port=6667 host.name=f-bcpc-vm3.bcpc.example.com advertised.host.name=f-bcpc-vm3.bcpc.example.com advertised.port=9092 … When I bring up Kafka, there is no process listening on port 9092 but Kafka

Re: migrating log data to new locations

2014-10-07 Thread Neha Narkhede
Is it possible to perform this migration without losing the data currently stored in the kafka cluster? Though I haven't tested this, the way this is designed should allow you to shut down a broker, move some partition directories over to the new log volume and restart the broker. You will have

Re: migrating log data to new locations

2014-10-07 Thread Steve Morin
Neha, I log volume or can it be volumes plural? -Steve On Tue, Oct 7, 2014 at 6:41 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Is it possible to perform this migration without losing the data currently stored in the kafka cluster? Though I haven't tested this, the way this is designed

Re: No process listening on advertised.port

2014-10-07 Thread Biju N
KAFKA-1092 (https://issues.apache.org/jira/browse/KAFKA-1092) provided the answer to the query. Thanks all. On Tue, Oct 7, 2014 at 8:26 AM, Biju N bijuatapa...@gmail.com wrote: Hello There, I have the following in my server.properties on a two node kafka test cluster …. port=6667

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-10-07 Thread Jun Rao
Otis, Yes, if you guys can help provide a patch in a few days, we can probably get it to the 0.8.2 release. Thanks, Jun On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi Jun, I think your MBean renaming approach will work. I see

Re: migrating log data to new locations

2014-10-07 Thread Jay Kreps
I think the more automated/lazy way right now would be to shutdown one broker, rm -rf all its data, add the data directories in config, and restart to let the broker restore off the replicas. This may actually be okay though it is a little slower. -Jay On Tue, Oct 7, 2014 at 3:25 PM, Jun Rao

rate of producers writing to a topic?

2014-10-07 Thread Josh J
Hi, Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the topic in particular ways depending on the incoming write throughput. Thanks, Josh

Re: rate of producers writing to a topic?

2014-10-07 Thread Josh J
Interested in the total number of tuples written per millisecond per topic. On Tue, Oct 7, 2014 at 3:56 PM, Josh J joshjd...@gmail.com wrote: Hi, Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the

Using MirrorMaker to move some data between clusters

2014-10-07 Thread István
Hi all, I have just a quick question. I was wondering if MirrorMaker is the right solution for this task, basically I need to move a subset of the data from a production cluster to a test cluster. My original idea was to set the log.retention.minutes to a low value like 12 hours so I could keep

Re: rate of producers writing to a topic?

2014-10-07 Thread Neil Harkins
On Tue, Oct 7, 2014 at 3:56 PM, Josh J joshjd...@gmail.com wrote: Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the topic in particular ways depending on the incoming write throughput. we've got a

Kafka AWS deployment + UI console

2014-10-07 Thread Shafaq
We are going to deploy Kafka in Production and also monitor it via console. (e.g. State of partitions in Broker- leader and slaves, state of consumers ) Is there out-of-the-box solution? What is the best and efficient way of deployment and monitoring Has someone tried this- looks promising

Re: rate of producers writing to a topic?

2014-10-07 Thread Josh J
we've got a cron which iterates each topic+partition and writes an index of timestamps-byte offset What is byte offset? Is this reliable to gauge the number of tuples being written to a topic when there are many different consumers writing to the same topic ? On Tue, Oct 7, 2014 at 4:35 PM,

Re: Kafka AWS deployment + UI console

2014-10-07 Thread Gwen Shapira
I'm using Hue's ZooKeeper app: http://gethue.com/new-zookeeper-browser-app/ This UI looks very cute, but I didn't try it yet: https://github.com/claudemamo/kafka-web-console Gwen On Tue, Oct 7, 2014 at 5:08 PM, Shafaq s.abdullah...@gmail.com wrote: We are going to deploy Kafka in Production

Re: Kafka AWS deployment + UI console

2014-10-07 Thread Jude K
We use http://quantifind.com/KafkaOffsetMonitor/... On Tue, Oct 7, 2014 at 8:49 PM, Gwen Shapira gshap...@cloudera.com wrote: I'm using Hue's ZooKeeper app: http://gethue.com/new-zookeeper-browser-app/ This UI looks very cute, but I didn't try it yet:

Re: Using MirrorMaker to move some data between clusters

2014-10-07 Thread Jun Rao
Yes, this is typical use case for mirror maker. Thanks, Jun On Tue, Oct 7, 2014 at 4:33 PM, István lecc...@gmail.com wrote: Hi all, I have just a quick question. I was wondering if MirrorMaker is the right solution for this task, basically I need to move a subset of the data from a