[
https://issues.apache.org/jira/browse/FLINK-24666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24666:
--------------------------------
Description:
In stream processing, records will be deleted when exceed state ttl (if
configured), and when the corresponding record's update arrives again, the
operator may not be able to handle it properly, we need a unified error
handling mechanism to handle this situation,
instead of each stateful operator currently handling its own.
e.g., currently, there's a lenient option which was not exposed to users on
RetractableTopNFunction
{quote}// flag to skip records with non-exist error instead to fail, true by
default.
private final boolean lenient = true
{quote}
So there's no chance to raise the exception when the record(s) unexpectedly
cleared by state ttl. Commonly this happens because a too shorter ttl at Line
190 or inconstancy between the two internal state(dataState and treeMap) at
other place.
{quote}List<RowData> inputs = dataState.get(key);
if (inputs == null)
Unknown macro: \{ // Skip the data if it's state is cleared because of state
ttl. if (lenient) Unknown macro}
else
Unknown macro: \{ throw new RuntimeException(STATE_CLEARED_WARN_MSG); }
}
{quote}
We'd better to expose it to users (default value can be true to keep consistent
with previous version) and should be unified to cover all the stateful stream
operators than specific to RetractableTopNFunction.
was:
In stream processing, records will be deleted when exceed state ttl (if
configured), and when the corresponding record's update arrives again, the
operator may not be able to handle it properly, we need a unified error
handling mechanism to handle this situation,
instead of each stateful operator currently handling its own.
e.g., currently, there's a lenient option which was not exposed to users on
RetractableTopNFunction
{quote}// flag to skip records with non-exist error instead to fail, true by
default.
private final boolean lenient = true
{quote}
So there's no chance to raise the exception when the record(s) unexpectedly
cleared by state ttl. Commonly this happens because a too shorter ttl at Line
190 or inconstancy between the two internal state(dataState and treeMap) at
other place.
{quote}List<RowData> inputs = dataState.get(key);
if (inputs == null)
Unknown macro: \{ // Skip the data if it's state is cleared because of state
ttl. if (lenient) Unknown macro}
else
Unknown macro: \{ throw new RuntimeException(STATE_CLEARED_WARN_MSG); }
}
{quote}
We'd better to expose it to users (default value can be true to keep consistent
with previous version)
> Add job level "table.exec.state-stale.error-handling" option and apply to
> related stateful stream operators
> -----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-24666
> URL: https://issues.apache.org/jira/browse/FLINK-24666
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Affects Versions: 1.14.0, 1.13.3
> Reporter: lincoln lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> In stream processing, records will be deleted when exceed state ttl (if
> configured), and when the corresponding record's update arrives again, the
> operator may not be able to handle it properly, we need a unified error
> handling mechanism to handle this situation,
> instead of each stateful operator currently handling its own.
> e.g., currently, there's a lenient option which was not exposed to users on
> RetractableTopNFunction
> {quote}// flag to skip records with non-exist error instead to fail, true by
> default.
> private final boolean lenient = true
> {quote}
> So there's no chance to raise the exception when the record(s) unexpectedly
> cleared by state ttl. Commonly this happens because a too shorter ttl at Line
> 190 or inconstancy between the two internal state(dataState and treeMap) at
> other place.
> {quote}List<RowData> inputs = dataState.get(key);
> if (inputs == null)
> Unknown macro: \{ // Skip the data if it's state is cleared because of state
> ttl. if (lenient) Unknown macro}
> else
> Unknown macro: \{ throw new RuntimeException(STATE_CLEARED_WARN_MSG); }
> }
> {quote}
> We'd better to expose it to users (default value can be true to keep
> consistent with previous version) and should be unified to cover all the
> stateful stream operators than specific to RetractableTopNFunction.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
