[
https://issues.apache.org/jira/browse/FLINK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284024#comment-17284024
]
Steven Zhen Wu edited comment on FLINK-21364 at 2/12/21, 10:57 PM:
-------------------------------------------------------------------
For pull based source (like file/Iceberg), it is probably more
natural/efficient to piggyback the `finishedSplitsIds` in the
`RequestSplitEvent`. A reader request a new split when the current split is
done.
It doesn't mean that a reader has to request for a new split when finishing
some splits, like bounded Kafka source case.
You have a good point that some sources (like Kafka/Kineses) may still need to
communicate the watermark info to coordinator/enumerator. In this case, it
definitely will be a separate type of event (like `WatermarkUpdateEvent`).
In our Iceberg source use cases, readers didn't actually report watermark. They
just need to report which split are finished. All the ordering/watermark
tracking is centralized in Iceberg source coordinator. But I can see that this
may not be a very generic scenario to change the `RequestSplitEvent` in
flink-runtime.
cc [~sundaram]
was (Author: stevenz3wu):
For pull based source (like file/Iceberg), it is probably more
natural/efficient to piggyback the `finishedSplitsIds` in the
`RequestSplitEvent`. A reader request a new split when the current split is
done.
It doesn't mean that a reader has to request for a new split when finishing
some splits, like bounded Kafka source case.
You have a good point that some sources (like Kafka/Kineses) may still need to
communicate the watermark info to coordinator/enumerator. In this case, it
definitely will be a separate type of event (like `WatermarkUpdateEvent`).
In our Iceberg source use cases, readers didn't actually report watermark. They
just need to report which split are finished. But I can see that this may not
be a very generic scenario to change the `RequestSplitEvent` in flink-runtime.
cc [~sundaram]
> piggyback finishedSplitIds in RequestSplitEvent
> -----------------------------------------------
>
> Key: FLINK-21364
> URL: https://issues.apache.org/jira/browse/FLINK-21364
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
> Affects Versions: 1.12.1
> Reporter: Steven Zhen Wu
> Priority: Major
> Labels: pull-request-available
>
> For some split assignment strategy, the enumerator/assigner needs to track
> the completed splits to advance watermark for event time alignment or rough
> ordering. Right now, `RequestSplitEvent` for FLIP-27 source doesn't support
> pass-along of the `finishedSplitIds` info and hence we have to create our own
> custom source event type for Iceberg source.
> Here is the proposal of add such optional info to `RequestSplitEvent`.
> {code}
> public RequestSplitEvent(
> @Nullable String hostName,
> @Nullable Collection<String> finishedSplitIds)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
