Hi Sachin, One option right now would be to precreate all internal topics in Kafka, and only after that start the Kafka Streams application. This would require you knowing the internal name of the topics (in this case you probably already know it, but I agree that in general this is a bit cumbersome).
Eno > On 8 Nov 2016, at 18:10, Sachin Mittal <[email protected]> wrote: > > Per message payload size. The basic question is how can I control the > internal change log topics parameters so as to avoid these errors. > > > On Tue, Nov 8, 2016 at 11:37 PM, R Krishna <[email protected]> wrote: > >> Are you talking about total messages and therefore size or per message >> payload size. >> >> On Tue, Nov 8, 2016 at 10:00 AM, Sachin Mittal <[email protected]> wrote: >> >>> Message size itself increases over the time. >>> >>> Message is something like >>> key=[list on objects] >>> >>> This increases with time and then at a point kafka is not able to add any >>> message to its topic because message size is greater than >>> max.message.bytes. >>> Since this is an internal topic based off a table I don't know how can I >>> control this topic. >>> >>> If I can set some retention.ms for this topic then I can purge old >>> messages >>> thereby ensuring that message size stays within limit. >>> >>> Thanks >>> Sachin >>> >>> >>> >>> On Tue, Nov 8, 2016 at 11:22 PM, Eno Thereska <[email protected]> >>> wrote: >>> >>>> Hi Sachin, >>>> >>>> Could you clarify what you mean by "message size increases"? Are >> messages >>>> going to the changelog topic increasing in size? Or is the changelog >>> topic >>>> getting full? >>>> >>>> Thanks >>>> Eno >>>> >>>>> On 8 Nov 2016, at 16:49, Sachin Mittal <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> We are using aggregation by key on a kstream to create a ktable. >>>>> As I read from >>>>> https://cwiki.apache.org/confluence/display/KAFKA/ >>>> Kafka+Streams%3A+Internal+Data+Management >>>>> it creates an internal changelog topic. >>>>> >>>>> However over the time the streaming application is run message size >>>>> increases and it starts throwing max.message.bytes exception. >>>>> >>>>> Is there a way to control the retention.ms time for internal >> changelog >>>>> topics so that messages are purged before they exceed this size. >>>>> >>>>> If not is there a way to control or avoid such an error. >>>>> >>>>> Thanks >>>>> Sachin >>>> >>>> >>> >> >> >> >> -- >> Radha Krishna, Proddaturi >> 253-234-5657 >>
