Hi Thomas, Thanks for your detailed reply for the design of Hybrid Source. I would reply to the questions you mentioned as follows:
1.Has there been related activity since the FLIP was created? Yes, the FLIP has the initial version of Hybrid Source implementation. You'd better refer to the repository: g...@github.com:wuchong/flink-hackathon.git . 2.The actual FLIP-27 source reader would need to signal to the "HybridSourceReader" (HSR) that they are done and then the HSR would send the switch event to the coordinator? The communication in Hybrid Source should be between "HybridSourceReader" and "HybridSplitEnumerator". After a "SourceReader" in "HybridSourceReader" finishes reading the source split assigned from split enumerator and switches to another "SourceReader", "HybridSourceReader" send a signal to "HybridSplitEnumerator" to notify that the "SourceReader" finishes reading. Then "HybridSplitEnumerator" receives the "SourceReaderSwitchedEvent" signal and switch to another "SplitEnumerator". 3.The actual split type that flows between enumerator and reader would be "HybridSourceSplit" and it would wrap the specific split (in the example either HDFS or Kafka)? "HybridSourceSplit" wraps the source split types for the "SourceSplit" created by the "SplitEnumerator" in "HybridSplitEnumerator". 4."HybridSplitEnumerator" still needs a way to extract them from the actual enumerator. That could be done by the enumerator checkpoint state mapping function looking at the current split assignments, which would not require modification of existing enumerators? IMO, the above way to extract them from the actual enumerator could be defined in the state conversion function which is invoked in "HybridSplitEnumerator" when switch one "SwitchableSplitEnumerator" to another "SwitchableSplitEnumerator". Because the "SwitchableSplitEnumerator" interface could return the end state of the actual enumerator. I don't know if it has answered your confusion for Hybrid Source. If you still have questions, keep discussion with me. Thanks for your attention. Best, Nicholas Jiang -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
