Edson Wang created FLINK-40396:
----------------------------------
Summary: Create SplitReaders on the split fetcher thread instead
of the caller thread
Key: FLINK-40396
URL: https://issues.apache.org/jira/browse/FLINK-40396
Project: Flink
Issue Type: Improvement
Reporter: Edson Wang
Follow-up promised in https://github.com/apache/flink-connector-kafka/pull/290
(FLINK-36434).
SplitFetcherManager currently invokes the split-reader supplier on the thread
that creates
the SplitFetcher (the source reader thread), not on the fetcher thread that
will use the
reader. Clients constructed inside a SplitReader therefore capture the wrong
thread at
construction time. For Kafka this meant the KafkaConsumer was created on a
thread it never
runs on afterwards; FLINK-36434 fixed it locally by making the consumer lazy
inside
KafkaPartitionSplitReader.
Deferring the supplier call itself onto the fetcher thread in
flink-connector-base would fix
this class of issue for all connectors, so no connector needs its own laziness
workaround.
Sketch: move the supplier invocation from SplitFetcher construction into the
fetcher
thread's startup (first run), while keeping construction failures propagating
through the
existing fetcher error-handling path (today a failing supplier throws
synchronously on the
caller thread — connectors relying on that would see the failure asynchronously
instead,
which needs to be called out in the compatibility notes).
I'm happy to work on this once assigned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
