[jira] [Commented] (KAFKA-6248) Enable configuration of internal topics of Kafka Streams applications
[ https://issues.apache.org/jira/browse/KAFKA-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818262#comment-16818262 ] Edmondo Porcu commented on KAFKA-6248: -- Does it work for max.request.size? -Dtopic.max.request.size=300 -Dproducer.max.request.size=300 Exception in thread "analysis-input-enricher-e1c4502d-62e9-43a6-bf59-b37dcfc77ce2-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: task [1_0] Abort sending since an error caught with a previous record (key NON PREVISTO value [B@62851189 timestamp 1555104485966) to topic analysis-input-enricher-cr-STATE-STORE-01-changelog due to org.apache.kafka.common.errors.RecordTooLargeException: The message is 1403267 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. > Enable configuration of internal topics of Kafka Streams applications > - > > Key: KAFKA-6248 > URL: https://issues.apache.org/jira/browse/KAFKA-6248 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Tim Van Laer >Priority: Minor > > In the current implementation of Kafka Streams, it is not possible to set > custom configuration to internal topics (e.g. max.message.bytes, > retention.ms...). It would be nice if a developer can set some specific > configuration. > E.g. if you want to store messages bigger than 1MiB in a state store, you > have to alter the corresponding changelog topic with a max.message.bytes > setting. > The workaround is to create the 'internal' topics upfront using the correct > naming convention so Kafka Streams will use the explicitly defined topics as > if they are internal. > An alternative is to alter the internal topics after the Kafka Streams > application is started and has created its internal topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6248) Enable configuration of internal topics of Kafka Streams applications
[ https://issues.apache.org/jira/browse/KAFKA-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264806#comment-16264806 ] Matthias J. Sax commented on KAFKA-6248: It is compatible: https://docs.confluent.io/current/streams/upgrade-guide.html#compatibility If this does what you need, would you mind close this as "no a problem"? > Enable configuration of internal topics of Kafka Streams applications > - > > Key: KAFKA-6248 > URL: https://issues.apache.org/jira/browse/KAFKA-6248 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Tim Van Laer >Priority: Minor > > In the current implementation of Kafka Streams, it is not possible to set > custom configuration to internal topics (e.g. max.message.bytes, > retention.ms...). It would be nice if a developer can set some specific > configuration. > E.g. if you want to store messages bigger than 1MiB in a state store, you > have to alter the corresponding changelog topic with a max.message.bytes > setting. > The workaround is to create the 'internal' topics upfront using the correct > naming convention so Kafka Streams will use the explicitly defined topics as > if they are internal. > An alternative is to alter the internal topics after the Kafka Streams > application is started and has created its internal topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6248) Enable configuration of internal topics of Kafka Streams applications
[ https://issues.apache.org/jira/browse/KAFKA-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264106#comment-16264106 ] Tim Van Laer commented on KAFKA-6248: - Hi Bill, Thank you for the information, I must have looked over it while browsing the docs. The solution you propose does suit my needs for now. Thanks! Now I just need to find out if my application running Kafka Streams 1.0.0 will be compatible with our 0.10.2 cluster. Regards, Tim > Enable configuration of internal topics of Kafka Streams applications > - > > Key: KAFKA-6248 > URL: https://issues.apache.org/jira/browse/KAFKA-6248 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Tim Van Laer >Priority: Minor > > In the current implementation of Kafka Streams, it is not possible to set > custom configuration to internal topics (e.g. max.message.bytes, > retention.ms...). It would be nice if a developer can set some specific > configuration. > E.g. if you want to store messages bigger than 1MiB in a state store, you > have to alter the corresponding changelog topic with a max.message.bytes > setting. > The workaround is to create the 'internal' topics upfront using the correct > naming convention so Kafka Streams will use the explicitly defined topics as > if they are internal. > An alternative is to alter the internal topics after the Kafka Streams > application is started and has created its internal topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6248) Enable configuration of internal topics of Kafka Streams applications
[
https://issues.apache.org/jira/browse/KAFKA-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260892#comment-16260892
]
Bill Bejeck commented on KAFKA-6248:
Hi [~timvanlaer]
Take a look at
[KIP-173|https://cwiki.apache.org/confluence/display/KAFKA/KIP-173%3A+Add+prefix+to+StreamsConfig+to+enable+setting+default+internal+topic+configs.]
included in Kafka Streams as of release 1.0, which allows you to configure
internal topics using the {{topic.}} prefix.
Note this is a global setting for all internal topics and the config needs to
be a valid property from {{org.apache.kafka.common.config.TopicConfig}}
HTH,
Bill
> Enable configuration of internal topics of Kafka Streams applications
> -
>
> Key: KAFKA-6248
> URL: https://issues.apache.org/jira/browse/KAFKA-6248
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Tim Van Laer
>Priority: Minor
>
> In the current implementation of Kafka Streams, it is not possible to set
> custom configuration to internal topics (e.g. max.message.bytes,
> retention.ms...). It would be nice if a developer can set some specific
> configuration.
> E.g. if you want to store messages bigger than 1MiB in a state store, you
> have to alter the corresponding changelog topic with a max.message.bytes
> setting.
> The workaround is to create the 'internal' topics upfront using the correct
> naming convention so Kafka Streams will use the explicitly defined topics as
> if they are internal.
> An alternative is to alter the internal topics after the Kafka Streams
> application is started and has created its internal topics.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
