Hi,

I'm building a custom source that reads from a distributed event queue,
similar to Pulsar or Kafka.

One interesting characteristic of this event queue is that some partitions
may temporarily become unavailable (i.e. a few hours). And it'll need to
resume from the previous progress when the partitions come online again, so
the state needs to be maintained.

The current solution is to suppress the error when reading from unavailable
partitions and keep recreating the client utilized by the split reader.
Apparently, this is not ideal because it is simply wasting resources on
doing nothing.

One potential alternative would be to use
`SplitEnumerator.handleSourceEvent` to signal the reader and temporarily
pause the fetcher somehow. Is this a feasible solution?

Best
Bin

Reply via email to