Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

I don't feel strongly on this, so I'm happy with whatever everyone else wants. Michael, I'm not arguing that people don't need to understand topologies, I just think it is like rocks db, you need to understand it when debugging/operating but not in the initial coding since the metaphor we're provi

Re: Capacity planning for Kafka Streams

Hi Damian, The rest of the logs were INFO messages about offset being committed. Anyways, the problem is resolved for now, after we increased the max.poll.interval.ms. For anyone else who is facing similar problem, please refer this thread. https://groups.google.com/forum/#!topic/confluent-platf

Relationship fetch.replica.max.bytes and message.max.bytes

Can fetch.replica.max.bytes be equal to message.max.bytes? 1. The defaults in the official Kafka documentation [1] have the parameter "fetch.replica.max.bytes" set to a higher value than "message.max.bytes". However, nothing in the description of these parameters implies that equality would be wro

Re: ISR churn

Sure…there are two types of purgatories: Consumer and Producer Consumer purgatory (for a partition leader) is a queue for pending requests for data (i.e. polling by some client for the respective partition). It’s basically a waiting area for poll requests. Generally speaking, the more consumer

Re: ISR churn

Hi David, I checked our cluster, the producer purgatory size is under 3 mostly. But I’m not quite understand this metrics, could you please explain it a little bit? Thanks, Jun > On Mar 22, 2017, at 3:07 PM, David Garcia wrote: > > producer purgatory size

Re: kafka streams in-memory Keyvalue store iterator remove broken on upgrade to 0.10.2.0 from 0.10.1.1

Hi, What I was trying to accomplish was the normal usage of the iterator interface to enable safe remove while iterating over a collection. I have used iterator.remove since kafka streams was released, so this has been the real functionality since release and in the absence of documentation to say

Re: ISR churn

Marcos, Radu, Are either of you seeing messages saying "Cached zkVersion [...] not equal to that in zookeeper"? If so, you may be hitting https://issues.apache.org/jira/browse/KAFKA-3042 Not sure if that helps you, since I haven't been able i

Re: clearing an aggregation?

I get that the windows are aligned along seconds but this doesn't really help with true clock alignment (IE top of the hour, midnight, etc). I can imagine a strategy using overlapping windows. One would (hypothetically) walk through the list until a window that spanned the desired time was found.

Re: ISR churn

Look at producer purgatory size. Anything greater than 10 is bad (from my experience). Keeping it under 4 seemed to help us. (i.e. if a broker is getting slammed with write, use rebalance tools or add a new broker). Also check network latency and/or adjust timeout for ISR checking. If on AW

Kafka Connect: Bootstrap broker [hostname] disconnected

Hello, Previously, we were using Kafka version 0.10.0.0 and Kafka Connect 0.10.0.0. Everything was working fine, but we would like to upgrade Connect to the latest version 0.10.1.1 to make use of some new features. We do not want to upgrade Kafka if we don't have to. However, it seems that the new

Kafka Connect: Bootstrap broker [hostname] disconnected

Hello, Previously, we were using Kafka version 0.10.0.0 and Kafka Connect 0.10.0.0. Everything was working fine, but we would like to upgrade Connect to the latest version 0.10.1.1 to make use of some new features. We do not want to upgrade Kafka if we don't have to. However, it seems that the new

Re: Error in running PageViewTypedDemo

Thanks for the info. With "page2",{"user":"2", "page":"22", "timestamp":143527817} as input for streams-pageview-input an "2",{"region":"CA","timestamp":143527817} as input for streams-userprofile-input, the following error is shown, Exception in thread "StreamThread-1" java.lang.Illegal

Need help determining consumer group offsets

I have a 0.8.2.2 cluster which has been configured with offsets.storage=kafka. We are experiencing some issues after a few nodes went down and wrong nodes were brought up in their place, fortunately not production. I am trying to determine what the current offsets are for a consumer group and can't

Re: ISR churn

Let me know if this fix your issue! I’d really interesting to know based on what information should we decide to expand the cluster- bytes per seconds or number of partitions on each broker? And what is the limitation. > On Mar 22, 2017, at 11:46 AM, Marcos Juarez wrote: > > We're seeing the

Re: Error in running PageViewTypedDemo

Thanks Michael. I am trying that out. What would be best way to clear the values entered for a given input topic. Due to typo, if incorrect values are entered in a topic, it has to be cleared. Otherwise, the program keeps failing at processing incorrect input value. On Wed, Mar 22, 2017 at 12:27 P

Re: kafka streams in-memory Keyvalue store iterator remove broken on upgrade to 0.10.2.0 from 0.10.1.1

To add to what Matthias said, in case the following isn't clear: - You should not (and, in 0.10.2, cannot any longer) call the iterator's remove() method, i.e. `KeyValueIterator#remove()` when iterating through a `KeyValueStore`. Perhaps this is something we should add to the `KeyValueIterator` j

Re: Error in running PageViewTypedDemo

IIRC the PageViewTypedDemo example requires input data where the username/userId is captured in the keys of messages/records, and further information in the values of those messages. The problem you are running into is that, when you are writing your input data via the console consumer, the record

Re: kafka streams in-memory Keyvalue store iterator remove broken on upgrade to 0.10.2.0 from 0.10.1.1

Hi, remove() should not be supported -- thus, it's actually a bug in 0.10.1 that got fixed in 0.10.2. Stores should only be altered by Streams and iterator over the stores should be read-only -- otherwise, you might mess up Streams internal state. I would highly recommend to reconsider the call

Re: ISR churn

To manually failover the controller, just delete the /controller znode in zookeeper On Wed, Mar 22, 2017 at 11:46 AM, Marcos Juarez wrote: > We're seeing the same exact pattern of ISR shrinking/resizing, mostly on > partitions with the largest volume, with thousands of messages per second. > It

Error in running PageViewTypedDemo

I have started exploring kafka streaming API. I'm trying to run PageViewTypedDemo program as it is without any changes locally on a desktop. Current kafka version is 0.10.1.0. With the following inputs from 2 different console, bin/kafka-console-producer.sh --broker-list localhost:9092 --topic s

Re: ISR churn

We're seeing the same exact pattern of ISR shrinking/resizing, mostly on partitions with the largest volume, with thousands of messages per second. It happens at least a hundred times a day in our production cluster. We do have hundreds of topics in our cluster, most of them with 20 or more partiti

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Regarding the naming of `StreamsTopologyBuilder` v.s. `StreamsBuilder` that are going to be used in DSL, I agree both has their arguments: 1. On one side, people using the DSL layer probably do not need to be aware (or rather, "learn about") of the "topology" concept, although this concept is a pu

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Forwarding to kafka-user. -- Forwarded message -- From: Michael Noll Date: Wed, Mar 22, 2017 at 8:48 AM Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API To: d...@kafka.apache.org Matthias, > @Michael: > > You seemed to agree with Jay about not exposing the `To

kafka streams in-memory Keyvalue store iterator remove broken on upgrade to 0.10.2.0 from 0.10.1.1

Hi, hope someone on kafka-streams team can help. Our application uses KeyValueIterator it = KeyValueStore.all(); ….. it.remove() This used to work but is now broken, causes our punctuate to fail and StreamThread to die. The cause seems to be that there were changes in 0.10.2.0 to InMemoryKe

Re: ISR churn

Any erros related to zookeeper seesion timeout? We can also check for excesssive GC. Some times this may due to forming multiple controllers due to soft failures. You can check ActiveControllerCount on brokers, only one broker in the cluster should have 1. Also check for network issues/partitions

ISR churn

Hello, Does anyone know how I can debug high ISR churn on the kafka leader on a cluster without traffic? I have 2 topics on a 4 node cluster (replica 4 and replica 3) and both show constant changes of the number of insync replicas: [2017-03-22 15:30:10,945] INFO Partition [__consumer_offsets,0]

Re: Capacity planning for Kafka Streams

Hi Mahendra, Are you able to share the complete logs? It is pretty hard to tell what is happening just from a few snippets of information. Thanks, Damian On Wed, 22 Mar 2017 at 12:16 Mahendra Kariya wrote: > To test Kafka streams on 0.10.2.0, we setup a new Kafka cluster with the > latest vers

Kafka running in VPN

Hello I am using Kafka-0.10.0 My Kafka brokers are running on a 3 node cluster in a VPN. The consumer and producer are not part of VPN; so I use ssh tunnels and update the bootstrap servers on the Kafka Client (consumer and producer) config accordingly. The Kafka Client is able to initiate the pri

Re: Commits of slow moving topics in Kafka Streams time out

Ok, yes increasing the retention minutes to something silly is indeed what we intend to do, and for now, that should work fine as we don't use many short lived consumers. For the long term, the change in semantics in the link you provide looks good to us. regards, Frank On Tue, Mar 21, 2017 at 8

Re: Capacity planning for Kafka Streams

To test Kafka streams on 0.10.2.0, we setup a new Kafka cluster with the latest version and used mirror maker to replicate the data from the 0.10.0.0 Kafka cluster. We pointed our streaming app to the newly created Kafka cluster. We have 5 nodes, each running the streaming app with 10 threads. In

Re: Kafka consumer offset reset

Hello Jakub, Maybe you're interested in feature that's not yet available but is being proposed and discussed (on dev ML) for future - see https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling Kind regards, Stevo Slavic. On Tue, Mar 21, 2017 at 4:43