[
https://issues.apache.org/jira/browse/FLINK-22781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369433#comment-17369433
]
JING ZHANG edited comment on FLINK-22781 at 6/25/21, 12:13 PM:
---------------------------------------------------------------
[~jark] Thanks a lot for reply.
I would like to summarize the current solution (without introducing extra
mechanism) to handle late UB message for window aggregate upon
changelog(contains update message).
User needs to set all the following 3 parameters:
(1) enable late fire by setting
{code:java}
table.exec.emit.late-fire.enabled : true{code}
(2) set per record emit behavior for late records by setting
{code:java}
table.exec.emit.late-fire.delay : 0 s
{code}
(3) keep window state for extra time after window is fired by setting
{code:java}
table.exec.emit.allow-latenss : 1 h
// 或者
table.exec.state.ttl: 1h{code}
The solution has two disadvantages:
(1) User may not realize that UB message maybe dropped as late event, so they
will not set related parameters.
(2) When use look for a solution to solve dropped UB message problem, current
solution is a bit of unconvinient for them because they need to set all the 3
parameters.
I agree with you provides recommended configurations in docs could help.
Besides, could we simplify the 3 parameters a little. for example, user only
need set allow-lateness, framework could atom set
`table.exec.emit.late-fire.enabled` to true and
set`table.exec.emit.late-fire.delay ` to 0s. Just like the behavior on
Datastream(user only need set allowedLateness on Windowedstream).
What do you think?
was (Author: qingru zhang):
[~jark] Thanks a lot for reply.
I would like to summarize the current solution (without introducing extra
mechanism) to handle late UB message for window aggregate upon
changelog(contains update message).
User needs to set all the following 3 parameters:
(1) enable late fire by setting
{code:java}
table.exec.emit.late-fire.enabled : true{code}
(2) set per record emit behavior for late records by setting
{code:java}
table.exec.emit.late-fire.delay : 0 s
{code}
(3) keep window state for extra time after window is fired by setting
{code:java}
table.exec.emit.allow-latenss : 1 h
// 或者
table.exec.state.ttl: 1h{code}
The solution has two disadvantages:
(1) User may not realize that UB message maybe dropped as late event, so they
will not set related parameters.
(2) When use look for a solution to solve dropped UB message problem, current
solution is a bit of unconvinient for them because they need to set all the 3
parameters.
I agree with you provides recommended configurations in docs could help.
Besides, could we simplify the 3 parameters a little. for example, user only
need set allow-lateness, framework could atom set
`table.exec.emit.late-fire.enabled` to true and
set`table.exec.emit.late-fire.delay ` to 0s.
What do you think?
> Incorrect result for group window aggregate when mini-batch is enabled
> ----------------------------------------------------------------------
>
> Key: FLINK-22781
> URL: https://issues.apache.org/jira/browse/FLINK-22781
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 1.14.0
> Reporter: godfrey he
> Assignee: JING ZHANG
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> We can reproduce this issue through adding the following code to
> {{GroupWindowITCase#testWindowAggregateOnUpsertSource}} method:
> {code:java}
> tEnv.getConfig.getConfiguration.setBoolean(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED, true)
> tEnv.getConfig.getConfiguration.set(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ALLOW_LATENCY,
> Duration.ofSeconds(1))
> tEnv.getConfig.getConfiguration.setLong(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_SIZE, 10L)
> {code}
> The reason is the group window without any data (the data may be retracted)
> should not send any record.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
