Re: ISR differs between Kafka Metadata and Zookeeper
I am seeing this behavior using librdkafka, as is another user. Listing the topic metadata with the tool provided with Kafka (kafka-topic.sh) shows all replicas in the ISR. However, using kafkacat[1] (built with librdkafka) shows that many ISRs are missing some replicas. I talked with Magnus (author of librdkafka), and he asked me to get a tcpdump of the response returned by the controller broker to kafkacat. The tcpdump showed that the broker itself was returning the ISRs missing some replicas. More information about this can be found here: https://github.com/edenhill/librdkafka/issues/147 This doesn't seem to be a transient problem, as the results returned by kafkacat are consistent. [1] https://github.com/edenhill/kafkacat On Thu, Oct 3, 2013 at 5:43 PM, Neha Narkhede neha.narkh...@gmail.com wrote: When you issue a metadata request to any broker, it responds with the cluster metadata in memory. The broker gets it through an UpdateMetadata request issued by the controller broker. If there are state changes in flight, there may be a delay in the UpdateMetadata request propagation to the broker serving the TopicMetadataRequest. However, this behavior is transient and in steady state TopicMetadataRequest should always return up-to-date information. Thanks, Neha On Thu, Oct 3, 2013 at 1:17 PM, Florian Weingarten f...@shopify.com wrote: Hi list, I am trying to debug a strange issue we are seeing. We are using Sarama [1], our own Go implementation of the Kafka API. Somehow, we either found a bug in Kafka, have a bug in our own code, or got our cluster into a weird state. If we use our library to query Kafka for metadata about any partition, it often (but not always!) returns an ISR that is a strict subset of the replica set. However, if I use the bin/kafka-list-topic.sh which comes with Kafka, it shows the two sets as equal. I believe that tool gets its data from Zookeeper. If I look at Zookeeper directly, I also see matching sets. Does anybody have any idea what could cause this? What's the most reliable way to get this data? Kafka or Zookeeper? Any idea why we are seeing this? And if it's not a bug in the reporting, what could have caused the replicas to not be in-sync (and how do we trigger them to catch up? They don't seem to do it automatically). Thanks! Flo [1]: https://github.com/Shopify/sarama
Re: Disactivating Yammer Metrics Monitoring
Hi Daniel, Thank you for your answer. It's possible that I didn't understood something, if so correct me please. From what I understood, from the kafka doc #monitoring http://kafka.apache.org/documentation.html#monitoring, kafka use Yammer Metrics for monitoring the servers (the brokers) and the clients (producers and consumers). Our web site is also using Yammer Metrics and push that to our Graphite server and our web site also produce message in kafka. From what I read, the Yammer Metrics GraphiteReporter is a kind of Singleton, once I Enable it, it is working for all the process. (But I might be wrong here...) We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade, kafka is monitoring in our Graphite Server and is hammering it, so we aren't able to use it because we always get timeout... SO, I was wondering if there is a way to disable the kafka monitoring to our Graphite server. We are using the code in the tag 0.8.1.1 on github, so if the kafka-ganglia isn't in the tag, we aren't using it :) François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure http://www.etsmtl.ca/ Capitaine Club Capra http://capra.etsmtl.ca/ VP-Communication - CS Games http://csgames.org 2014 Jeux de Génie http://www.jdgets.com/ 2011 à 2014 Magistrat Fraternité du Piranha http://fraternitedupiranha.com/ Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton d...@danielcompton.net wrote: Hi Francois I didn't quite understand how you've set up your metrics reporting. Are you using the https://github.com/criteo/kafka-ganglia metrics reporter? If so then you should be able to adjust the config to exclude the metrics you don't want, with kafka.ganglia.metrics.exclude.regex. On 18 September 2014 07:55, François Langelier f.langel...@gmail.com wrote: Hi all! We are using yammer metrics to monitor some parts of our system. Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data getting in our graphite server and from what I saw, it looks like it all come from our producers. From what i understand, since we already use graphite, our graphiteReporter is enable in our main web site and our kafka producers are having fun using it too to monitor in graphite. The problem is that right now kafka is hammering of graphite server and we have difficulty to saw our monitored data... Is there a way to deactivate the monitoring of our kafka producers? François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure http://www.etsmtl.ca/ Capitaine Club Capra http://capra.etsmtl.ca/ VP-Communication - CS Games http://csgames.org 2014 Jeux de Génie http://www.jdgets.com/ 2011 à 2014 Magistrat Fraternité du Piranha http://fraternitedupiranha.com/ Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
subscribe kafka mail list
Hi, I am interested in Kafka, I want to subscribe kafka mail lists, Thanks.
Re: subscribe kafka mail list
http://kafka.apache.org/contact.html François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure http://www.etsmtl.ca/ Capitaine Club Capra http://capra.etsmtl.ca/ VP-Communication - CS Games http://csgames.org 2014 Jeux de Génie http://www.jdgets.com/ 2011 à 2014 Magistrat Fraternité du Piranha http://fraternitedupiranha.com/ Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior On Fri, Sep 19, 2014 at 4:14 AM, happyyan...@gmail.com happyyan...@gmail.com wrote: Hi, I am interested in Kafka, I want to subscribe kafka mail lists, Thanks.
Copying messages from a single partition topic to a multi-partition topic
Hi, We have an interesting problem to solve due to a very large traffic volumes on particular topics. In our initial system configuration we had only one partition per topic, and in in a couple of topics we have built up huge backlogs of several million messages that our consumers are slowly processing. However, now that we have this constant backlog, we wish to repartition those topics into several partitions, and allow parallel consumers to run to handle the high message volume. If we simply repartition the topic, say from 1 to 4 partitions, the backlogged messages stay in partition 1, while partitions 2,3,4 only get newly arrived messages. To eat away the backlog, we need to redistribute the backlogged messages evenly among the 4 partitions. The tools I've seen do not allow me to rewrite or replay the existing backlogged messages from one partition into the same or another topic with several partitions. - using kafka.tools.MirrorMaker does not allow me to move the data within the same zookeeper network, and - using kafka.tools.ReplayLogProducer does not write to multiple partitions. It seems that it will write only from a single partition to a single partition. Does anyone have any other way to solve this problem or a better way of using the kafka tools? Thanks Dennis
Re: Copying messages from a single partition topic to a multi-partition topic
I would look at writing a service that reads from your existing topic and writes to a new topic with (e.g. four) partitions. You will also need to pay attention to the partitioning policy (or implement your own), as the default hashing in the current kafka version default can lead to poor distribution. Best Regards, -Jonathan On Sep 19, 2014, at 8:57 AM, Dennis Haller dhal...@talemetry.com wrote: Hi, We have an interesting problem to solve due to a very large traffic volumes on particular topics. In our initial system configuration we had only one partition per topic, and in in a couple of topics we have built up huge backlogs of several million messages that our consumers are slowly processing. However, now that we have this constant backlog, we wish to repartition those topics into several partitions, and allow parallel consumers to run to handle the high message volume. If we simply repartition the topic, say from 1 to 4 partitions, the backlogged messages stay in partition 1, while partitions 2,3,4 only get newly arrived messages. To eat away the backlog, we need to redistribute the backlogged messages evenly among the 4 partitions. The tools I've seen do not allow me to rewrite or replay the existing backlogged messages from one partition into the same or another topic with several partitions. - using kafka.tools.MirrorMaker does not allow me to move the data within the same zookeeper network, and - using kafka.tools.ReplayLogProducer does not write to multiple partitions. It seems that it will write only from a single partition to a single partition. Does anyone have any other way to solve this problem or a better way of using the kafka tools? Thanks Dennis
Re: Copying messages from a single partition topic to a multi-partition topic
Agree. Why not write a service that reads from existing topics and writes to new ones. Point existing producers to the new partitions wait for the service to finish reading. Since you have only one partition right now, I'd assume you don't have any partitioning logic per se. Think about that since with multiple partitions you'd want your messages distributed evenly. On Fri, Sep 19, 2014 at 9:32 PM, Jonathan Weeks jonathanbwe...@gmail.com wrote: I would look at writing a service that reads from your existing topic and writes to a new topic with (e.g. four) partitions. You will also need to pay attention to the partitioning policy (or implement your own), as the default hashing in the current kafka version default can lead to poor distribution. Best Regards, -Jonathan On Sep 19, 2014, at 8:57 AM, Dennis Haller dhal...@talemetry.com wrote: Hi, We have an interesting problem to solve due to a very large traffic volumes on particular topics. In our initial system configuration we had only one partition per topic, and in in a couple of topics we have built up huge backlogs of several million messages that our consumers are slowly processing. However, now that we have this constant backlog, we wish to repartition those topics into several partitions, and allow parallel consumers to run to handle the high message volume. If we simply repartition the topic, say from 1 to 4 partitions, the backlogged messages stay in partition 1, while partitions 2,3,4 only get newly arrived messages. To eat away the backlog, we need to redistribute the backlogged messages evenly among the 4 partitions. The tools I've seen do not allow me to rewrite or replay the existing backlogged messages from one partition into the same or another topic with several partitions. - using kafka.tools.MirrorMaker does not allow me to move the data within the same zookeeper network, and - using kafka.tools.ReplayLogProducer does not write to multiple partitions. It seems that it will write only from a single partition to a single partition. Does anyone have any other way to solve this problem or a better way of using the kafka tools? Thanks Dennis
Re: ISR differs between Kafka Metadata and Zookeeper
This may be due to https://issues.apache.org/jira/browse/KAFKA-1367 On Thu, Oct 03, 2013 at 04:17:45PM -0400, Florian Weingarten wrote: Hi list, I am trying to debug a strange issue we are seeing. We are using Sarama [1], our own Go implementation of the Kafka API. Somehow, we either found a bug in Kafka, have a bug in our own code, or got our cluster into a weird state. If we use our library to query Kafka for metadata about any partition, it often (but not always!) returns an ISR that is a strict subset of the replica set. However, if I use the bin/kafka-list-topic.sh which comes with Kafka, it shows the two sets as equal. I believe that tool gets its data from Zookeeper. If I look at Zookeeper directly, I also see matching sets. Does anybody have any idea what could cause this? What's the most reliable way to get this data? Kafka or Zookeeper? Any idea why we are seeing this? And if it's not a bug in the reporting, what could have caused the replicas to not be in-sync (and how do we trigger them to catch up? They don't seem to do it automatically). Thanks! Flo [1]: https://github.com/Shopify/sarama
Re: How to use kafka as flume source.
Just to update (better late than never!): The Kafka source sink for Flume were updated to latest Kafka version and improved a bit (offsets are now committed after data is written to Flume channel). If you build Flume from trunk, you'll get these. Gwen On Sun, Aug 3, 2014 at 10:31 AM, Andrew Ehrlich ehrlic...@gmail.com wrote: The author may or may not have upgraded the project to be compatible with 0.8.1. You could message the author directly to ask. On Sun, Aug 3, 2014 at 6:53 AM, rafeeq s rafeeq.ec...@gmail.com wrote: Thanks. But above link example uses flume-ng https://github.com/apache/flume to communicate with *kafka 0.7,2 http://kafka.apache.org/07/quickstart.html *. I am using *kafka 0.8.1*. Please advice an example of flume-ng with kafka 0.8.1. Thanks in advance Regards, Rafeeq S *(“What you do is what matters, not what you think or say or plan.” )* On Sun, Aug 3, 2014 at 4:40 PM, Sharninder sharnin...@gmail.com wrote: https://github.com/baniuyao/flume-ng-kafka-source On Sun, Aug 3, 2014 at 6:15 PM, rafeeq s rafeeq.ec...@gmail.com wrote: Hi, We are planning to use kafka as *flume source*. Please advice me, how to use kafka as source in flume. please share if there is any best example of *flume- kafka source- hdfs sink*. Regards, Rafeeq S *(What you do is what matters, not what you think or say or plan. )*
How to use RPC mechanism in Kafka?
Hi, Please tell me how to use request/response mechanism in kafka? Thanks Lavish Goel
Re: Handling errors in the new (0.8.2) Java Client's Producer
Hello Andrew,
I think you would want a sync producer for your use case? You can try to
call get() on the returned metadata future of the send() call instead of
using a callback; the pattern is something like:
for (message in messages)
producer.send(message).get()
The get() call will block until the message response has received, and will
throw an exception if the response is failure, you the then catch the
exception and pause the producer.
Guozhang
On Thu, Sep 18, 2014 at 6:30 PM, Andrew Stein
andrew.st...@quantumretail.com wrote:
I am trying to understand the best practices for working with the new
(0.8.2) Producer interface.
We have a process in a large server that writes a lot of data to Kafka.
However, this data is not mission critical. When a problem arises writing
to Kafka, most specifically network issues, but also full Producer buffers,
we want the server to continue working, but to stop sending data to Kafka,
allowing other tasks to continue. The issue I have is handling messages
that have been sent to the producer but are waiting to go to Kafka. These
messages remain long after my processing is over, timing out, writing to
the logs, and
preventing me from moving forward. I am looking for some way to tell the
client to stop forwarding messages to Kafka.
This is what I have so far:
class ErrorCallback implements Callback {
@Override
public void onCompletion(RecordMetadata metadata, Exception
exception) {
if (exception == null) { // The message was sent,
return;
}
stopProducerSendAndClose();
String threadName = Thread.currentThread().getName();
if (!threadName.equals(kafka-producer-network-thread)) { //
Some of the callbacks happen on my thread
} else { // We are in KafkaProducer's ioThread == commit
suicide.
Thread.currentThread().interrupt();
throw new ThreadDeath(); // Cannot throw an Exception as is
will just be caught and logged.
}
}
}
My question is, is this the correct approach, or is there some other way to
stop sending messages (short of going synced).
Andrew Stein
--
-- Guozhang
Re: How to use RPC mechanism in Kafka?
Do you mean that you want to know the protocol? https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Guozhang On Fri, Sep 19, 2014 at 12:18 PM, lavish goel lavis...@gmail.com wrote: Hi, Please tell me how to use request/response mechanism in kafka? Thanks Lavish Goel -- -- Guozhang
Re: Handling errors in the new (0.8.2) Java Client's Producer
Hi Guozhang,
I specifically to not want to do a get() on every future. This is
equivalent to going synced, which is a performance killer for us.
What I am looking for is some way to tell a producer to forget about queued
messages. I have done some more testing on my solution and it needs some
work. That said, the reliance on a thread named
kafka-producer-network-thread is something I don't like -- too much
knowledge of the KafkaProducer's internals.
Hoping for other suggestions...
Andrew Stein
On Fri, Sep 19, 2014 at 2:31 PM, Guozhang Wang wangg...@gmail.com wrote:
Hello Andrew,
I think you would want a sync producer for your use case? You can try to
call get() on the returned metadata future of the send() call instead of
using a callback; the pattern is something like:
for (message in messages)
producer.send(message).get()
The get() call will block until the message response has received, and will
throw an exception if the response is failure, you the then catch the
exception and pause the producer.
Guozhang
On Thu, Sep 18, 2014 at 6:30 PM, Andrew Stein
andrew.st...@quantumretail.com wrote:
I am trying to understand the best practices for working with the new
(0.8.2) Producer interface.
We have a process in a large server that writes a lot of data to Kafka.
However, this data is not mission critical. When a problem arises writing
to Kafka, most specifically network issues, but also full Producer
buffers,
we want the server to continue working, but to stop sending data to
Kafka,
allowing other tasks to continue. The issue I have is handling messages
that have been sent to the producer but are waiting to go to Kafka.
These
messages remain long after my processing is over, timing out, writing to
the logs, and
preventing me from moving forward. I am looking for some way to tell the
client to stop forwarding messages to Kafka.
This is what I have so far:
class ErrorCallback implements Callback {
@Override
public void onCompletion(RecordMetadata metadata, Exception
exception) {
if (exception == null) { // The message was sent,
return;
}
stopProducerSendAndClose();
String threadName = Thread.currentThread().getName();
if (!threadName.equals(kafka-producer-network-thread)) { //
Some of the callbacks happen on my thread
} else { // We are in KafkaProducer's ioThread == commit
suicide.
Thread.currentThread().interrupt();
throw new ThreadDeath(); // Cannot throw an Exception as
is
will just be caught and logged.
}
}
}
My question is, is this the correct approach, or is there some other way
to
stop sending messages (short of going synced).
Andrew Stein
--
-- Guozhang
Re: Handling errors in the new (0.8.2) Java Client's Producer
Andrew
If you see BaseProducer.scala, there's the following code snippet
override def send(topic: String, key: Array[Byte], value: Array[Byte]) {
val record = new ProducerRecord(topic, key, value)
if(sync) {
this.producer.send(record).get()
} else {
this.producer.send(record,
new ErrorLoggingCallback(topic, key, value, false))
}
}
NewShinyProducer is used in ProducerPerformance.scala. When I tested both
sync mode and async mode, I didn't see much difference. So, I guess you can
safely use sync.
Also, there's ProducerPerformance.java in the new version of producer.
Callback cb = stats.nextCompletion(sendStart, payload.length,
stats);
producer.send(record, cb);
public Callback nextCompletion(long start, int bytes, Stats stats) {
Callback cb = new PerfCallback(this.iteration, start, bytes,
stats);
this.iteration++;
return cb;
}
private static final class PerfCallback implements Callback {
private final long start;
private final int iteration;
private final int bytes;
private final Stats stats;
public PerfCallback(int iter, long start, int bytes, Stats stats) {
this.start = start;
this.stats = stats;
this.iteration = iter;
this.bytes = bytes;
}
public void onCompletion(RecordMetadata metadata, Exception
exception) {
long now = System.currentTimeMillis();
int latency = (int) (now - start);
this.stats.record(iteration, latency, bytes, now);
if (exception != null)
exception.printStackTrace();
}
}
I hope this will be helpful.
On Fri, Sep 19, 2014 at 1:47 PM, Andrew Stein
andrew.st...@quantumretail.com wrote:
Hi Guozhang,
I specifically to not want to do a get() on every future. This is
equivalent to going synced, which is a performance killer for us.
What I am looking for is some way to tell a producer to forget about queued
messages. I have done some more testing on my solution and it needs some
work. That said, the reliance on a thread named
kafka-producer-network-thread is something I don't like -- too much
knowledge of the KafkaProducer's internals.
Hoping for other suggestions...
Andrew Stein
On Fri, Sep 19, 2014 at 2:31 PM, Guozhang Wang wangg...@gmail.com wrote:
Hello Andrew,
I think you would want a sync producer for your use case? You can try to
call get() on the returned metadata future of the send() call instead of
using a callback; the pattern is something like:
for (message in messages)
producer.send(message).get()
The get() call will block until the message response has received, and
will
throw an exception if the response is failure, you the then catch the
exception and pause the producer.
Guozhang
On Thu, Sep 18, 2014 at 6:30 PM, Andrew Stein
andrew.st...@quantumretail.com wrote:
I am trying to understand the best practices for working with the new
(0.8.2) Producer interface.
We have a process in a large server that writes a lot of data to Kafka.
However, this data is not mission critical. When a problem arises
writing
to Kafka, most specifically network issues, but also full Producer
buffers,
we want the server to continue working, but to stop sending data to
Kafka,
allowing other tasks to continue. The issue I have is handling messages
that have been sent to the producer but are waiting to go to Kafka.
These
messages remain long after my processing is over, timing out, writing
to
the logs, and
preventing me from moving forward. I am looking for some way to tell
the
client to stop forwarding messages to Kafka.
This is what I have so far:
class ErrorCallback implements Callback {
@Override
public void onCompletion(RecordMetadata metadata, Exception
exception) {
if (exception == null) { // The message was sent,
return;
}
stopProducerSendAndClose();
String threadName = Thread.currentThread().getName();
if (!threadName.equals(kafka-producer-network-thread)) {
//
Some of the callbacks happen on my thread
} else { // We are in KafkaProducer's ioThread == commit
suicide.
Thread.currentThread().interrupt();
throw new ThreadDeath(); // Cannot throw an Exception
as
is
will just be caught and logged.
}
}
}
My question is, is this the correct approach, or is there some other
way
to
stop sending messages (short of going synced).
Andrew Stein
--
-- Guozhang
Re: Disactivating Yammer Metrics Monitoring
Hi, I don't have any source or configs handy to check things, but you are saying you've configured Kafka to use GraphiteReporter, right? So why not remove that config, so metrics stop being sent to Graphite if your Graphite setup is suffering? If you do that and you still want to see your Kafka metrics, you can always use SPM http://sematext.com/spm/ for Kafka (though some of the graphs will be empty until we get KAFKA-1481, or something else that improves metrics, in). If you just want to use it temporarily, just use the free 30-day trial version until you beef up your Graphite setup. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Fri, Sep 19, 2014 at 10:08 AM, François Langelier f.langel...@gmail.com wrote: Hi Daniel, Thank you for your answer. It's possible that I didn't understood something, if so correct me please. From what I understood, from the kafka doc #monitoring http://kafka.apache.org/documentation.html#monitoring, kafka use Yammer Metrics for monitoring the servers (the brokers) and the clients (producers and consumers). Our web site is also using Yammer Metrics and push that to our Graphite server and our web site also produce message in kafka. From what I read, the Yammer Metrics GraphiteReporter is a kind of Singleton, once I Enable it, it is working for all the process. (But I might be wrong here...) We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade, kafka is monitoring in our Graphite Server and is hammering it, so we aren't able to use it because we always get timeout... SO, I was wondering if there is a way to disable the kafka monitoring to our Graphite server. We are using the code in the tag 0.8.1.1 on github, so if the kafka-ganglia isn't in the tag, we aren't using it :) François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure http://www.etsmtl.ca/ Capitaine Club Capra http://capra.etsmtl.ca/ VP-Communication - CS Games http://csgames.org 2014 Jeux de Génie http://www.jdgets.com/ 2011 à 2014 Magistrat Fraternité du Piranha http://fraternitedupiranha.com/ Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton d...@danielcompton.net wrote: Hi Francois I didn't quite understand how you've set up your metrics reporting. Are you using the https://github.com/criteo/kafka-ganglia metrics reporter? If so then you should be able to adjust the config to exclude the metrics you don't want, with kafka.ganglia.metrics.exclude.regex. On 18 September 2014 07:55, François Langelier f.langel...@gmail.com wrote: Hi all! We are using yammer metrics to monitor some parts of our system. Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data getting in our graphite server and from what I saw, it looks like it all come from our producers. From what i understand, since we already use graphite, our graphiteReporter is enable in our main web site and our kafka producers are having fun using it too to monitor in graphite. The problem is that right now kafka is hammering of graphite server and we have difficulty to saw our monitored data... Is there a way to deactivate the monitoring of our kafka producers? François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure http://www.etsmtl.ca/ Capitaine Club Capra http://capra.etsmtl.ca/ VP-Communication - CS Games http://csgames.org 2014 Jeux de Génie http://www.jdgets.com/ 2011 à 2014 Magistrat Fraternité du Piranha http://fraternitedupiranha.com/ Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
Re: Interaction of retention settings for broker and topic plus partitions
That's right. The rule is that a log segment is deleted if either the size or the time limit is reached. Log sizes are per partition. Thanks, Jun On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson gp...@keen.io wrote: Hello all! I'm curious about the interaction of server and topic level retention settings. It's not clear to me the precedence of the follow: - broker's default log.retention.bytes - topic's retention.bytes (which defaults to broker's log.retention.bytes) - broker's log.retention.hours and log.retention.minutes (if both are specified then it seems to be the lower of the two, since it's when either is exceeded) It seems that the rule is that when any of these are violated then the log segment is deleted. Is this right? Also, just to be clear: The log sizes in questions are for a single partitions logs? I have a situation where my per-topic retention.bytes is very high, but my default log.retention.hours is lower (the default @ 168 hours). It seems that it's truncating at the log.retention.hours instead of the topic's retention.bytes. Am I understanding this correctly? :) -- Cory Watson Principal Infrastructure Engineer // Keen IO
Re: How to use RPC mechanism in Kafka?
Thank you for your response. I want to implement request/response type model. For eg. I have a producer that publish a message(He wants to know some status) to topic. A consumer pulls the message and processes the request. Now I want that, the response of this request should go that producer. Can you please tell me how can I implement this. Thanks Lavish On Sat, Sep 20, 2014 at 1:26 AM, Guozhang Wang wangg...@gmail.com wrote: Do you mean that you want to know the protocol? https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Guozhang On Fri, Sep 19, 2014 at 12:18 PM, lavish goel lavis...@gmail.com wrote: Hi, Please tell me how to use request/response mechanism in kafka? Thanks Lavish Goel -- -- Guozhang
