Hi, tison and Jark,
Thanks for your advice.
Different from Flink CDC connectors, InLong's changes to MySQL CDC
connector mainly involve entire database migration and support for rich
metadata fields (like canal), which are very important for Apache InLong to
provide external services as data integration. For example, for the entire
database migration, if one schema is used for one table, it will be a waste
of resources and difficult to maintain, and it will also put pressure on
the upstream database.We also particularly hope to work with Flink CDC to
see how to support the entire database migration and support richer
metadata fields, so that we can directly rely on it. We don't know if Flink
CDC has plans and technical solutions for these two scenarios. We can talk
about it together or contribute directly.
In addition, in order to speed up discussion and follow-up cooperation,
I have created a WeChat group [1], and interested contributors can join the
direct discussion.
Thanks again.
[1]
[image: image.png]
Jark Wu <[email protected]> 于2022年7月23日周六 22:45写道：
> Thanks for starting this discussion, tison!
>
> It's great to see InLong is using flink-cdc-connectors to extract data from
> various databases.
> I think tison raised a good point about contributing back instead of
> forking. I believe this can
> help the InLong community in the long term by leveraging improvements and
> bug fixes from
> upstream. Flink CDC can also benefit from more contributions from the
> InLong community.
> I believe it's the only way that we can improve collaboration between
> different OSS projects.
>
> In a word, I highly appreciate learning the missing part of Flink CDC that
> InLong needs.
> We can discuss the missing features here or in the issues.
>
> Cheers,
> Jark
>
>
> On Fri, 22 Jul 2022 at 23:56, tison <[email protected]> wrote:
>
> > Hi,
> >
> > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> > Flink CDC mysql connector[2].
> >
> > With a short discussion with Charles I learned that InLong developers
> > added metadata supports and migration of a whole database.
> >
> > It could be better to see if InLong community can contribute such
> supports
> > back to Flink CDC upstream, and avoid maintaining a whole fork as well as
> > periodically catch up upstream updates.
> >
> > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> >
> > Perhaps we can start with creating an issue on Flink CDC repo to describe
> > the case, contribute back the improvements, and switch dependency to the
> > upstream.
> >
> > Best,
> > tison.
> >
> > [1]
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > [2]
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >
> >
>
--
Best wishes,
Charles Zhang
