sean-rossignol commented on PR #13283: URL: https://github.com/apache/kafka/pull/13283#issuecomment-1809404175
Hey, anyway I could help unblock this pr @vamossagar12 ? We have an internal kstreams app with a wildcard consumer and one bottleneck we're hitting is as the number of source topics grows so does our initial total heap size when we need to rehydrate the components down stream of our cooker. The problem is that we set the max heap size across all instances of the kstreams app to handle the initial expected load when we first deploy the application. Over time the number of source topics grow and come rehydrate time we will have to increase the resources dedicated to the instance, futz with the buffered.records.per.partition config, or reset the offsets of the source topic partitions in batches, otherwise all instances of our application will go OOM. Being able to define and enforce a maximum input buffer size at the instance/thread level would allow us to handle these rehydration events without needing to change any other elements of our deployments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
