[
https://issues.apache.org/jira/browse/FLINK-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-8106:
---------------------------
Description:
Currently the logic of {{AbstractKeyedCEPPatternOperator}} is as follows when
event time enabled:
1) When element comes, buffer it in {{MapState}} and and register a timer at
{{watermark + 1}}
2) When event timer triggered, scan the {{MapState}} and find the elements
below current watermark and process it. If there are remaining elements to
process or the NFA is not empty, register a new timer at {{watermark + 1}}.
Let's assume that watermark comes about 5 seconds later than the event on
average, then we will scan about 5000 times the {{MapState}} before processing
the events. We find that most of the cpu is wasted serialization and
deserialization the {{MapState}} when profiling an CEP use case. After make the
optimization of the timer logic, the throughput increases from {{10+}} tps to
about {{3500}} tps for one operator for RocksDBStateBackend.
was:
Currently the logic of {{AbstractKeyedCEPPatternOperator}} is as follows when
event time enabled:
1) When element comes, buffer it in {{MapState}} and and register a timer at
{{watermark + 1}}
2) When event timer triggered, scan the {{MapState}} and find the elements
below current watermark and process it. If there are remaining elements to
process or the NFA is not empty, register a new timer at {{watermark + 1}}.
Let's assume that watermark comes about 5 seconds later than the event on
average, then we will scan about 5000 times the {{MapState}} before processing
the events. We find that most of the cpu is wasted serialization and
deserialization the {{MapState}} when profiling an CEP use case. After make the
optimization of the timer logic, the throughput increases from {{10+}} tps to
about {{3500}} tps for one operator.
> Optimize the timer logic in AbstractKeyedCEPPatternOperator
> -----------------------------------------------------------
>
> Key: FLINK-8106
> URL: https://issues.apache.org/jira/browse/FLINK-8106
> Project: Flink
> Issue Type: Bug
> Components: CEP
> Reporter: Dian Fu
> Assignee: Dian Fu
>
> Currently the logic of {{AbstractKeyedCEPPatternOperator}} is as follows when
> event time enabled:
> 1) When element comes, buffer it in {{MapState}} and and register a timer at
> {{watermark + 1}}
> 2) When event timer triggered, scan the {{MapState}} and find the elements
> below current watermark and process it. If there are remaining elements to
> process or the NFA is not empty, register a new timer at {{watermark + 1}}.
> Let's assume that watermark comes about 5 seconds later than the event on
> average, then we will scan about 5000 times the {{MapState}} before
> processing the events. We find that most of the cpu is wasted serialization
> and deserialization the {{MapState}} when profiling an CEP use case. After
> make the optimization of the timer logic, the throughput increases from
> {{10+}} tps to about {{3500}} tps for one operator for RocksDBStateBackend.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
