Hi, all I'd like to follow up on our proposal regarding the integration of Flink Materialized Table refresh tasks (DSIP-90). We've had some initial discussions on both the mailing list and the GitHub issue, but we're hoping to gather more community input before moving forward.
For those who may have missed the earlier discussions, we propose adding support for refreshing Flink Materialized Tables (introduced in Flink 1.20) within DolphinScheduler through a new task type. This would enable better integration between DolphinScheduler and Flink Materialized Tables. Some technical points that have been discussed on GitHub: 1. For job status monitoring, we plan to use a single task instance that monitors the job until completion by leveraging the operation handle (containing jobId and clusterInfo) obtained after submitting a refresh request. 2. Our current implementation focuses specifically on scheduled batch jobs rather than streaming jobs. The proposal details are available at: https://github.com/apache/dolphinscheduler/issues/17086 We welcome any additional feedback, questions, or suggestions from the community. Your input is valuable in ensuring this integration meets the needs of DolphinScheduler users. Best regards, Feng Jin On Wed, Apr 9, 2025 at 8:42 PM Feng Jin <[email protected]> wrote: > Hi, all > > I'd like to kindly follow up on our proposal regarding the integration of > Flink Materialized Table refresh tasks into DolphinScheduler (DSIP-90). > > We've received some positive initial feedback (thank you Chufeng for the > +1), but we're hoping to gather more community input before proceeding > further. The proposal aims to enhance DolphinScheduler's integration with > Flink 1.20's Materialized Tables feature, which would benefit users working > with both systems. > > For those who haven't had a chance to review, the proposal is available at: > https://github.com/apache/dolphinscheduler/issues/17086 > > We welcome any thoughts, concerns, or suggestions on this integration. > Your input will be valuable in shaping this feature to best serve the > community's needs. > > Best regards, > Feng Jin > > > On Tue, Apr 8, 2025 at 12:10 AM Chufeng Gao <[email protected]> wrote: > >> Hi Feng, >> >> Thanks for bringing this up! Glad to see better integration with >> Apache DolphinScheduler and Apache Flink. >> >> Since both projects are open-source, it is possible for us to have >> automated integration tests to cover this new plugin so that it will >> not increase the burden of maintenance for the community. >> >> I'm +1 to this. >> >> Best Regards, >> >> Chufeng (Eric) Gao >> >> >> >> On Wed, Mar 26, 2025 at 5:16 PM Feng Jin <[email protected]> wrote: >> > >> > Dear DolphinScheduler Development Community, >> > >> > I hope this message finds you well. >> > >> > Ron(cc'd) and I would like to initiate a discussion regarding the >> > integration of Flink Materialized Table refresh tasks into >> DolphinScheduler. >> > >> > >> > Flink introduced Materialized Tables[1] in version 1.20, and we would >> like >> > to propose adding support for refreshing these tables within >> > DolphinScheduler through a new task type. >> > >> > >> > *Background:* >> > >> > Materialized Tables rely on scheduled refresh jobs to keep data >> updated. To >> > enable this functionality in DolphinScheduler, we propose creating a new >> > task type for executing Flink refresh jobs. This would provide a >> seamless >> > way to integrate Flink Materialized Table refresh operations into our >> > existing data pipeline workflows. >> > >> > >> > >> > *What’s Needed:* >> > >> > Introduce a new task type in DolphinScheduler to periodically trigger >> the >> > refresh of Flink Materialized Tables. >> > >> > >> > >> > I have opened an issue to track further discussions and gather community >> > feedback: [https://github.com/apache/dolphinscheduler/issues/17086] >> > >> > Looking forward to your thoughts and suggestions. >> > >> > >> > >> > >> > [1]. >> > >> https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/ >> > >> > >> > >> > Best regards, >> > >> > Feng Jin >> >
