[ https://issues.apache.org/jira/browse/KAFKA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stanislav Savulchik updated KAFKA-9355: --------------------------------------- Attachment: metric-removal.log > RocksDB statistics are removed from MBeans when EOS enabled and empty local > state dir > ------------------------------------------------------------------------------------- > > Key: KAFKA-9355 > URL: https://issues.apache.org/jira/browse/KAFKA-9355 > Project: Kafka > Issue Type: Bug > Components: metrics, streams > Affects Versions: 2.4.0 > Reporter: Stanislav Savulchik > Priority: Major > Attachments: metric-removal.log > > > *Steps to Reproduce* > Set processing.guarantee = exactly_once and remove local state dir in order > to force state restoration from changelog topics. > *Expected Behavior* > There are registered MBeans like > kafka.streams:type=stream-state-metrics,client-id=<application.id>-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=<state.store> > for persistent RockDB KeyValueStore-s after streams task state restoration. > *Actual Behavior* > There are no registered MBeans like above after streams task state > restoration. > *Details* > I managed to inject custom MetricsReporter in order to log metricChange and > metricRemoval calls. According to the logs at some point the missing metrics > are removed and never restored later. Here is an excerpt for > number-open-files metric: > {noformat} > 16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName > [name=number-open-files, group=stream-state-metrics, description=Number of > currently open files, > tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1, > task-id=0_0, rocksdb-state-id=buffered-event}] > 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics > recording trigger > 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Adding statistics for store buffered-event of > task 0_0 > 16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread > [morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No > checkpoint found for task 0_0 state store buffered-event changelog > morpheus.conversion-buffered-event-changelog-0 with EOS turned on. > Reinitializing the task and restore its state from the beginning. > 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Removing statistics for store buffered-event of > task 0_0 > 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Removing metrics recorder for store > buffered-event of task 0_0 from metrics recording trigger > 16:33 > 16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName > [name=number-open-files, group=stream-state-metrics, description=Number of > currently open files, > tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1, > task-id=0_0, rocksdb-state-id=buffered-event}] > 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics > recording trigger > 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics > Recorder for buffered-event] Adding statistics for store buffered-event of > task 0_0 > ... > (no more calls to metricChange for the removed number-open-files > metric){noformat} > -- This message was sent by Atlassian Jira (v8.3.4#803005)