[
https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henry Cai updated KAFKA-3595:
-----------------------------
Description:
Currently state store replication always go through a compact kafka topic. For
some state stores, e.g. JoinWindow, there are no duplicates in the store, there
is not much benefit using a compacted topic.
The problem of using compacted topic is the records can stay in kafka broker
forever. In my use case, my key is ad_id, it's incrementing all the time, not
bounded, I am worried the disk space on broker for that topic will go forever.
I think we either need the capability to purge the compacted records on broker,
or allow us to specify different compact option for state store replication.
was:Add the ability to record metrics in the serializer/deserializer
components. As it stands, I cannot record latency/sensor metrics since the API
does not provide the context at the serde levels. Exposing the ProcessorContext
at this level may not be the solution; but perhaps change the configure method
to take a different config or init context and make the StreamMetrics available
in that context along with config information.
> Add capability to specify replication compact option for stream store
> ---------------------------------------------------------------------
>
> Key: KAFKA-3595
> URL: https://issues.apache.org/jira/browse/KAFKA-3595
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.10.1.0
> Reporter: Henry Cai
> Assignee: Guozhang Wang
> Priority: Minor
>
> Currently state store replication always go through a compact kafka topic.
> For some state stores, e.g. JoinWindow, there are no duplicates in the store,
> there is not much benefit using a compacted topic.
> The problem of using compacted topic is the records can stay in kafka broker
> forever. In my use case, my key is ad_id, it's incrementing all the time,
> not bounded, I am worried the disk space on broker for that topic will go
> forever.
> I think we either need the capability to purge the compacted records on
> broker, or allow us to specify different compact option for state store
> replication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
