Hi Gustavo,
thank you for this excellent design doc and coming up with all these
different combinations of use cases. These two PTFs will significantly
improve the Flink CDC story and help polishing the stream/table duality
story.
+1 for this proposal.
Naming-wise I could also imagine calling it TO_CDC/FROM_CDC. But the
current naming fits well to other API endpoints and Flink terminology,
which is why I support FROM_CHANGELOG/TO_CHANGELOG.
Cheers,
Timo
On 06.02.26 18:52, Gustavo de Morais wrote:
Hi everyone,
I'd like to propose FLIP-564: Support *FROM_CHANGELOG* and *TO_CHANGELOG*
built-in PTFs [1] for discussion.
Flink's DataStream API offers flexible methods like toChangelogStream() and
fromChangelogStream() to work with changelog streams, but SQL users
currently lack this capability. This FLIP introduces two built-in Process
Table Functions (PTFs) to bring *similar* functionality with additional
features to Flink SQL:
- *FROM_CHANGELOG*: Converts an append-only stream of CDC records into a
dynamic table, enabling, for example, custom CDC connector implementations
directly in SQL.
- *TO_CHANGELOG*: Converts a dynamic table back into an append-only
changelog stream - the first operator that makes it possible to convert a
retract/upsert stream back to append in SQL.
Both PTFs support flexible operation mapping (e.g., Debezium-style 'c',
'u', 'd' codes), before/after image handling, configurable state TTL, and
watermark-based ordering. They are designed to work symmetrically, so
FROM_CHANGELOG(TO_CHANGELOG(table)) round-trips correctly.
The naming follows the existing DataStream API convention. Alternative
names were considered (e.g., ENCODE_CHANGELOG, DEMATERIALIZE) - they are
under the rejected alternatives. If you have any better suggestions, feel
free to share them here.
Looking forward to your feedback and thoughts.
Kind regards,
Gustavo de Morais
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-564%3A+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs
