[
https://issues.apache.org/jira/browse/KAFKA-6556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
igor mazor updated KAFKA-6556:
------------------------------
Description:
Allowing to attach callbacks in kafka streams, to be triggered when a window is
expired,
would help in use cases when the final state of the window is required.
It would be also useful if together with that functionally the user would be
able to control whether the callback would be triggered in addition to emitting
the normal change log down the stream, or only triggering the callback when the
window is expired. (for example in cases when only the final window state is
required, and any updates to the window state during the window time interval
are not important)
An example for use case could be left join with proper sql semantics:
A left join B currently would emit (a1,null) and (a1,b1) if b1 arrives within
the defined join time window.
what I would like is to have ONLY ONE result:
(a1,null) if no b1 arrived during the defined join time window, OR ONLY
(a1,b1) if b1 did arrived in the specified join time window.
One possible solution could be to use the current kafka streams left join with
downstream processor which would put the results in a windowed Ktable.
The window size would be same as for the left join operation, however, only
the final state of that window would be emitted down the stream once the time
window is expired.
So if the left join produces (a1, null) and after X minutes no (a1, b1) was
produced, eventually only (a1, null) would be emitted, on the other hand, if
the left join produces (a1, null) and after X-t minutes (a1, b1) was produced
by the left join operation => only (a1, b1) would be emitted eventually down
the stream after X minutes.
Another use case is when the window state is written to another kafka topic
which is then used to persist the window states into a db, However, many times
only the final window state
is required, and functionality to get only the last window state would help in
reducing load from the db, since only the final window state would be persisted
to the db, instead of multiple db writes for each window state update.
was:
Allowing to attach callbacks in kafka streams, to be triggered when a window is
expired,
would help in use cases when the final state of the window is required.
It would be also useful if together with that functionally the user would be
able to control whether the callback would be triggered in addition to emitting
the normal change log down the stream, or only triggering the callback when the
window is expired. (for example in cases when only the final window state is
required, and any updates to the window state during the window time interval
are not important)
An example for use case could be left join with proper sql semantics:
A left join B currently would emit (a1,null) and (a1,b1) if b1 arrives within
the defined join time window.
what I would like is to have ONLY ONE result:
(a1,null) if no b1 arrived during the defined join time window, OR ONLY (a1,b1)
if b1 did arrived in the specified join time window.
One possible solution could be to use the current kafka streams left join with
downstream processor which would put the results in a windowed Ktable.
The window size would be same as for the left join operation, however, only the
final state of that window would be emitted down the stream once the time
window is expired.
So if the left join prodces (a1, null) and after X minutes no (a1, b1) was
produced, eventually only (a1, null) would be emitted, on the other hand, if
the left join prduces (a1, null) and after X-t minutes (a1, b1) was produced by
the left join operation => only (a1, b1) would be emitted eventually down the
stream after X minutes.
Another use case is when the window state is written to another kafka topic
which is then used to persist the window states into a db, However, many times
only the final window state
is required, and functionality to get only the last window state would help in
reducing load from the db, since only the final window state would be persisted
to the db, instead of multiple db writes for each window state update.
> allow to attach callbacks in kafka streams, to be triggered when a window is
> expired
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-6556
> URL: https://issues.apache.org/jira/browse/KAFKA-6556
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: igor mazor
> Priority: Major
>
> Allowing to attach callbacks in kafka streams, to be triggered when a window
> is expired,
> would help in use cases when the final state of the window is required.
> It would be also useful if together with that functionally the user would be
> able to control whether the callback would be triggered in addition to
> emitting the normal change log down the stream, or only triggering the
> callback when the window is expired. (for example in cases when only the
> final window state is required, and any updates to the window state during
> the window time interval are not important)
> An example for use case could be left join with proper sql semantics:
> A left join B currently would emit (a1,null) and (a1,b1) if b1 arrives
> within the defined join time window.
> what I would like is to have ONLY ONE result:
> (a1,null) if no b1 arrived during the defined join time window, OR ONLY
> (a1,b1) if b1 did arrived in the specified join time window.
> One possible solution could be to use the current kafka streams left join
> with downstream processor which would put the results in a windowed Ktable.
> The window size would be same as for the left join operation, however, only
> the final state of that window would be emitted down the stream once the time
> window is expired.
> So if the left join produces (a1, null) and after X minutes no (a1, b1) was
> produced, eventually only (a1, null) would be emitted, on the other hand, if
> the left join produces (a1, null) and after X-t minutes (a1, b1) was produced
> by the left join operation => only (a1, b1) would be emitted eventually down
> the stream after X minutes.
>
> Another use case is when the window state is written to another kafka topic
> which is then used to persist the window states into a db, However, many
> times only the final window state
> is required, and functionality to get only the last window state would help
> in reducing load from the db, since only the final window state would be
> persisted to the db, instead of multiple db writes for each window state
> update.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
