Regarding kafka memory not distributed across cluster

Hi all, We are having a query regarding memory consumption on kafka scale out. It would be very helpful if you can give suggestion/solution for the below query. We are running kafka as docker container on kubernetes. Memory limit of 4GiB is configured for Kafka broker POD. With some large load

Re: InvalidStateStoreException on KStream join using GlobalKtables

Hello Ishita, Is it consistently reproducing? And which Kafka version are you using? Guozhang On Thu, May 2, 2019 at 5:24 PM Ishita Rakshit wrote: > Hi, > I have a Kafka Streams application where I am joining a KStream that reads > from "topic1" with a GlobalKTable that reads from "topic2"

Re: Changing tumbling windows inclusion

Hi John, on Slack Matthias suggested to have my own transform to window the data myself, I'll have a look into it and the Windows implementation as you suggested and see what I can do! Thanks for the advice! -- Alessandro Tagliapietra On Tue, May 7, 2019 at 8:45 AM John Roesler wrote: > Hi

Re: Changing tumbling windows inclusion

Hi Alessandro, Interesting. I agree, messing with the record timestamp to achieve your goal sounds too messy. It should be pretty easy to plug in your own implementation of Windows, instead of using the built-in TimeWindows, if you want slightly different windowing behavior. Does that work for

KSQL CLI: cannot execute script with SELECT query (but can execute other scripts)

Hi, one more question on KSQL CLI: using "run script /myScript.ksql" commands I can run scripts with "CREATE STREAM ..." and "CREATE TABLE ..." commands - no problem. However, when I try to run a script with a SELECT statement, I'm getting an error: Script: top3_reqtime.ksql SELECT

Confluent Center: KSQL query shows less results than when running from KSQL CLI

Hi! I'm running the same query from KSQL CLI and from the Confluent Control Center (KSQL Development). I set 'auto.offset.reset' = 'latest'; to only use latest events. Once I send 20 test events - I get full (correct) results from this query in KSQL CLI - but only partial results in the Control

Re: Mirror Maker tool is not running

Shallow iteration should be enabled for the consumer, not producer. Ryanne On Mon, May 6, 2019, 8:02 AM ASHOK MACHERLA wrote: > Dear Team Members > > > > Please find these below configurations for mirror maker tool scripts > > > > *source-cluster.config* > > > >

LogCleaner is not removing Transaction Records

Hi, We are using Kafka Streams with exactly-once enabled on a Kafka cluster for a while. Recently we found that the size of __consumer_offsets partitions grew huge. Some partition went over 30G. This caused Kafka to take quite long to load "__consumer_offsets" topic on startup (it loads the

Re: Best Practice Scaling Consumers

Hi Morritz - I don’t believe the number of Kafka consumers is restricted to the number of partitions. When you create a topic - and indicate both the number of partitions and a key - it causes your key value pairs to be allocated to a specific partition on the basis of a hash function on the

Setting log.dir to a tmpfs

Hello, I'm currently running a Kafka Cluster with a retention policy of 30mins and 10min log segments. In practice, this uses less than 20GB of file system space. Thus, I've been thinking about having no persistent storage at all. In the worst case, I do not mind losing the last 30min of

RE: Mirror Maker tool is not running

Dear Senthil In that links , there is no proper information about to fix this issue. Could you please tell me what configuration required in source and target sides Sent from Mail for Windows 10 From:

Re: Mirror Maker tool is not running

Looks like you are hitting : https://issues.apache.org/jira/browse/KAFKA-6947 or https://jira.apache.org/jira/browse/KAFKA-6177 --Senthil On Mon, May 6, 2019 at 6:32 PM ASHOK MACHERLA wrote: > Dear Team Members > > > > Please find these below configurations for mirror maker tool scripts > > >

Changing tumbling windows inclusion

Hello everyone, I'm trying to window a stream of machine production data, this use case needs a message with timestamp ending at the tumbling window end to be included in the current window not the next, because the message production amount refers to the previous x seconds. This doesn't work