Re: Mirror Maker

It is possible to implement offset-preserving mirrors - however, that would work only if there is one source cluster mirroring into target cluster(s) (as opposed to mirroring multiple source clusters into one target cluster). Anyway, as Jun said right now you have to either read from the tail or he

How far off is branch 0.8 to -0.8.0-beta1-candidate1? Is branch 0.8 stable enough to be used in production?

We are looking to adopt Kafka for our in stream processing use case. One of the issue seems to be that we use scala 2.10.2 however the the 0.8 Beta1 release does not seem to support scala 2.10.x. As a result, we are not able to use Kafka Scala client. I was wondering if we can use branch 0.8 which

Metrics Reporter not Valid

These warnings are fine - we should think about how to get rid of them. I think the issue is that there's a single bag of properties that is passed to the server and to the metrics reporter. The server instantiation process will will ignore these properties but will warn about properties that it do

Re: Mirror Maker

Tom, MirrorMaker is nothing but a tool with an embedded consumer and producer. So, mirroring is one-way. The offsets btw the source and the target clusters are not preserved though. If the consumer is real time, during failover, you can start the consumer in the failover site by consuming from the

Re: full disk

Yes, manually removing the old log files is the simplest solution right now. Thanks, Jun On Mon, Sep 23, 2013 at 9:16 AM, Paul Mackles wrote: > Done: > > https://issues.apache.org/jira/browse/KAFKA-1063 > > Out of curioisity, is manually removing the older log files the only > option at this

Re: Metrics Reporter not Valid

even using the default version of server.properties file, after start I got these messages: [2013-09-24 10:24:35,587] WARN Property kafka.csv.metrics.reporter.enabled is not valid (kafka.utils.VerifiableProperties) [2013-09-24 10:24:35,588] WARN Property kafka.metrics.reporters is not valid (kafka

Re: Overzealous logging?

Agree that this is annoying. We plan to fix this largely as part of the Client Rewrite project. If you have ideas on fixing it before that, could you file a JIRA where we can start a discussion? Thanks, Neha On Mon, Sep 23, 2013

Re: Kafka consumer - Mbean for max lag

On the consumer side, look for "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager". Updated the website to reflect that. Thanks, Neha On Mon, Sep 23, 2013 at 12:48 PM, Rajasekar Elango wrote: > In kafka documentation for > monitoring

Mirror Maker

Hi All, I am hoping that mirror maker is what I'm looking for. I would like to have complete data center fail over from one kafka cluster to another. If one data center goes down, my producers and consumers will start using the mirrored cluster until the primary is back online. Is that something I

Re: Understanding java options used by Kafka

Thanks Neha, it does help. I am running tests without those settings, and I'll run them again with it with gc monitoring. On Sep 23, 2013 12:02 PM, "Neha Narkhede" wrote: > I'm not sure about the reason for -Djava.awt.headless=true, but the GC > options were picked after quite a lot of GC tuning

Overzealous logging?

Sorry for the crazy long log trace here (feel free to ignore this message :)) I'm just wondering if there's an easy way to sensibly reduce the amount of logging that a kafka produer (0.8) will emit if I try to send a message (with ack level 1), if no broker is currently running? This is from one

Kafka consumer - Mbean for max lag

In kafka documentation for monitoring. I see "we should be looking at max in messages among all partitions.". All I can see is mbeans *kafka.server".FetcherLagMetrics.*ConsumerFetcherThread* and it's value is pretty much 0. Is this the correct

Re: full disk

I just encountered the same issue (and I ended up following the same work-around as Paul). One thing I noticed too, is that since the broker went down hard with an IOException when the disk filled up, it also needed 'recover' most of the logs on disk as part of the startup sequence. So any log-cl

Re: full disk

Done: https://issues.apache.org/jira/browse/KAFKA-1063 Out of curioisity, is manually removing the older log files the only option at this point? From: Paul Mackles mailto:pmack...@adobe.com>> To: "users@kafka.apache.org" mailto:users@kafka.apache.org>> Subject:

Re: Understanding java options used by Kafka

I'm not sure about the reason for -Djava.awt.headless=true, but the GC options were picked after quite a lot of GC tuning for the client/server configuration that LinkedIn has deployed in production. This works well when producers are deployed with request.required.acks=1 and consumers with fetch.w

Re: How far off is branch 0.8 to -0.8.0-beta1-candidate1? Is branch 0.8 stable enough to be used in production?

We fixed quite a few critical bugs after the beta1 release and we are preparing for releasing 0.8-final in roughly 3 weeks or so. We are operating 0.8 HEAD in production at LinkedIn but you could wait for the 0.8-final release if you think 0.8 HEAD is too risky. Thanks, Neha On Mon, Sep 23, 2013

Re: Metrics Reporter not Valid

no, I am using Kafka 0.8.0-beta1. On Mon, Sep 23, 2013 at 10:19 PM, Neha Narkhede wrote: > The metrics reporter was introduced in 0.8. Are you depending on Kafka > 0.7.x? > > Thanks, > Neha > On Sep 23, 2013 5:34 AM, "Henry Ma" wrote: > > > hi, > > > > In config/server.properties I added some p

How far off is branch 0.8 to -0.8.0-beta1-candidate1? Is branch 0.8 stable enough to be used in production?

We are looking to adopt Kafka for our in stream processing use case. One of the issue seems to be that we use scala 2.10.2 however the the 0.8 Beta1 release does not seem to support scala 2.10.x. As a result, we are not able to use Kafka Scala client. I was wondering if we can use branch 0.8 which

Re: How far off is branch 0.8 to -0.8.0-beta1-candidate1? Is branch 0.8 stable enough to be used in production?

I've been building the 0.8 branch and running tests in a semi-production environment and haven't had any troubles yet. I had one problem resulting from KAFKA-1046 where the Annotations*.scala files all were loaded by our custom build system, resulting in duplicate class declarations. We had to

Re: Metrics Reporter not Valid

The metrics reporter was introduced in 0.8. Are you depending on Kafka 0.7.x? Thanks, Neha On Sep 23, 2013 5:34 AM, "Henry Ma" wrote: > hi, > > In config/server.properties I added some properties for metrics report > using ganglia reporter( https://github.com/criteo/kafka-ganglia ): > > kafka.me

Re: Heartbeat btw producer and broker

You can configure the producer to reconnect to the brokers and set the reconnect interval to less than an hour. The config that controls this is reconnect.time.interval.ms. Thanks, Neha On Sep 23, 2013 12:14 AM, "Rhapsody" wrote: > Hi everyone, > I'm using Kafka 0.7.2 > > My firewall forcely clo

Metrics Reporter not Valid

hi, In config/server.properties I added some properties for metrics report using ganglia reporter( https://github.com/criteo/kafka-ganglia ): kafka.metrics.reporters=com.criteo.kafka.KafkaGangliaMetricsReporter kafka.ganglia.metrics.reporter.enabled=true kafka.ganglia.metrics.host=localhost kafka

Heartbeat btw producer and broker

Hi everyone, I'm using Kafka 0.7.2 My firewall forcely close the tcp session when it's no transmission btw two end points for one hour. When a producer in that network dosen't send any message to Kafka broker for one hour, it makes problem. I can't touch that firewall configuration. Alternative