Re: compression-rate-avg returns 0 with compression enabled

Yes, please. Ismael On 15 Dec 2015 11:52, "tao xiao" wrote: > Hi team, > > I found that the producer metric compression-rate-avg always returns 0 even > with compression.type set to snappy. I drilled down the code and discovered > that the position of bytebuffer in >

Consumer offset not reset to smallest when auto.offset.reset=smallest

We are noticing into these errors against certain partitions (4 out of 12 partitions) on the consumer and keeps filling up the logs: ERROR consumer.ConsumerFetcherThread: [ConsumerFetcherThread-], Current offset 104088851 for partition [some-topic,5] out of range; reset offset to 104088851

0.8.2 high level consumer with one-time-use group.id's?

When using the 0.8.2 high level consumer, what is the impact of creating many one-time use groupIds and checkpointing offsets using those? I have a use case where upon every boot, I want to consume an entire topic from the very beginning, all partitions. We are using the high level consumer for

how to programatically monitor Kafka availability

We want to be able to monitor the ability to send messages to Kafka topics. We want to be aware of the inability to do so before the time we attempt to send a message. What we're looking for is something like a heartbeat. The reason we need this is that in our deployment environment, Kafka

Re: Consumer offset not reset to smallest when auto.offset.reset=smallest

I also notice that the errors are for the partitions that seem to be under replicated. We got 3 brokers and one of the brokers never seems to be chosen as leader for any of the 12 partitions of the topic. On Tue, Dec 15, 2015 at 11:24 AM, Buntu Dev wrote: > We are noticing

Re: Low-latency, high message size variance

Hey Jens, The purpose of pause() is to stop fetches for a set of partitions. This lets you continue calling poll() to send heartbeats. Also note that poll() generally only blocks for rebalances. In code, something like this is what I was thinking: while (running) { ConsumerRecords

Re: Kafka Producer 0.9 performance issue with small messages

Dave, This is a new app being developed to use the Bluemix Messaging Hub 0.9 beta service. No migration involved from 0.8. With batching working for us (having fixed our own coding bug), we’re quite pleased with the numbers we’re seeing. Our mindset is that in using a cloud service solution,

compression-rate-avg returns 0 with compression enabled

Hi team, I found that the producer metric compression-rate-avg always returns 0 even with compression.type set to snappy. I drilled down the code and discovered that the position of bytebuffer in org.apache.kafka.common.record.Compressor is reset to 0 by RecordAccumulator.drain() before calling

Re: Per Topic Metrics

i found the problem by myself. since i don't know nothing about JMX and the Beans, i was querying the bean wrong. i still don't know how to query the specific topic metrics in jconsole. but it works for me to query (with jolokia) instead of: kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics

Re: Kafka 0.9 consumer API question

Hey Rajiv, I agree the Set/List inconsistency is a little unfortunate (another annoying one is pause() which uses a vararg). I think we should probably add the following variants: assign(Collection) subscribe(Collection) pause(Collection) I can open a JIRA to fix this. As for returning the

Kafka 0.9 consumer API question

We are trying to use the Kafka 0.9 consumer API to poll specific partitions. We consume partitions based on our own logic instead of delegating that to Kafka. One of our use cases is handling a change in the partitions that we consume. This means that sometimes we need to consume additional

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

We had to revert to 0.8.3 because three of our topics seem to have gotten corrupted during the upgrade. As soon as we did the upgrade producers to the three topics I mentioned stopped being able to do writes. The clients complained (occasionally) about leader not found exceptions. We restarted our

Re: Kafka 0.9 consumer API question

Hey Rajiv, My point was that you could maintain the assignment set yourself in a field, which would eliminate the need to copy the set returned by assignment(). Then it's just one copy to convert it to a list, and we can fix this by adding the assign() variant I suggested above. By the way,

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Hey Rajiv, Are you using snappy compression? On Tue, Dec 15, 2015 at 12:52 PM, Rajiv Kurian wrote: > We had to revert to 0.8.3 because three of our topics seem to have gotten > corrupted during the upgrade. As soon as we did the upgrade producers to > the three topics I

Re: Kafka 0.9 consumer API question

Hi Jason, The copying is not a problem in terms of performance. It's just annoying to write the extra code. My point with the copy is that since the client is already making a copy when it returns the set to me, why would it matter if I modify the copy. Creating an unmodifiable set on top of a

Re: Kafka 0.9 consumer API question

Right I could do that. Thanks for creating the JIRA! On Tue, Dec 15, 2015 at 3:01 PM, Jason Gustafson wrote: > Hey Rajiv, > > My point was that you could maintain the assignment set yourself in a > field, which would eliminate the need to copy the set returned by >

Re: Low-latency, high message size variance

I was talking with Jay this afternoon about this use case. The tricky thing about adding a ping() or heartbeat() API is that you have to deal with the potential for rebalancing. This means either allowing it to block while a rebalance completes or having it raise an exception indicating that a

Re: Low-latency, high message size variance

Hi Jason, Thanks for your response. See replies inline: On Tuesday, December 15, 2015, Jason Gustafson wrote: > Hey Jens, > > I'm not sure I understand why increasing the session timeout is not an > option. Is the issue that there's too much uncertainly about processing >

Re: compression-rate-avg returns 0 with compression enabled

created https://issues.apache.org/jira/browse/KAFKA-2993. I will submit a patch for this On Wed, 16 Dec 2015 at 00:53 Ismael Juma wrote: > Yes, please. > > Ismael > On 15 Dec 2015 11:52, "tao xiao" wrote: > > > Hi team, > > > > I found that the producer