Hi, all I'd like to kindly follow up on our proposal regarding the integration of Flink Materialized Table refresh tasks into DolphinScheduler (DSIP-90).
We've received some positive initial feedback (thank you Chufeng for the +1), but we're hoping to gather more community input before proceeding further. The proposal aims to enhance DolphinScheduler's integration with Flink 1.20's Materialized Tables feature, which would benefit users working with both systems. For those who haven't had a chance to review, the proposal is available at: https://github.com/apache/dolphinscheduler/issues/17086 We welcome any thoughts, concerns, or suggestions on this integration. Your input will be valuable in shaping this feature to best serve the community's needs. Best regards, Feng Jin On Tue, Apr 8, 2025 at 12:10 AM Chufeng Gao <[email protected]> wrote: > Hi Feng, > > Thanks for bringing this up! Glad to see better integration with > Apache DolphinScheduler and Apache Flink. > > Since both projects are open-source, it is possible for us to have > automated integration tests to cover this new plugin so that it will > not increase the burden of maintenance for the community. > > I'm +1 to this. > > Best Regards, > > Chufeng (Eric) Gao > > > > On Wed, Mar 26, 2025 at 5:16 PM Feng Jin <[email protected]> wrote: > > > > Dear DolphinScheduler Development Community, > > > > I hope this message finds you well. > > > > Ron(cc'd) and I would like to initiate a discussion regarding the > > integration of Flink Materialized Table refresh tasks into > DolphinScheduler. > > > > > > Flink introduced Materialized Tables[1] in version 1.20, and we would > like > > to propose adding support for refreshing these tables within > > DolphinScheduler through a new task type. > > > > > > *Background:* > > > > Materialized Tables rely on scheduled refresh jobs to keep data updated. > To > > enable this functionality in DolphinScheduler, we propose creating a new > > task type for executing Flink refresh jobs. This would provide a seamless > > way to integrate Flink Materialized Table refresh operations into our > > existing data pipeline workflows. > > > > > > > > *What’s Needed:* > > > > Introduce a new task type in DolphinScheduler to periodically trigger the > > refresh of Flink Materialized Tables. > > > > > > > > I have opened an issue to track further discussions and gather community > > feedback: [https://github.com/apache/dolphinscheduler/issues/17086] > > > > Looking forward to your thoughts and suggestions. > > > > > > > > > > [1]. > > > https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/ > > > > > > > > Best regards, > > > > Feng Jin >
