I agree with Steven. This logic can be added in a dynamic config framework that can bind into Flink operators. We probably don't need to let Flink runtime handle it.
On Fri, Jun 4, 2021 at 8:11 AM Steven Wu <stevenz...@gmail.com> wrote: > I am not sure if we should solve this problem in Flink. This is more like > a dynamic config problem that probably should be solved by some > configuration framework. Here is one post from google search: > https://medium.com/twodigits/dynamic-app-configuration-inject-configuration-at-run-time-using-spring-boot-and-docker-ffb42631852a > > On Fri, Jun 4, 2021 at 7:09 AM 刘建刚 <liujiangangp...@gmail.com> wrote: > >> Hi everyone, >> >> Flink jobs are always long-running. When the job is running, users >> may want to control the job but not stop it. The control reasons can be >> different as following: >> >> 1. >> >> Change data processing’ logic, such as filter condition. >> 2. >> >> Send trigger events to make the progress forward. >> 3. >> >> Define some tools to degrade the job, such as limit input qps, >> sampling data. >> 4. >> >> Change log level to debug current problem. >> >> The common way to do this is to stop the job, do modifications and >> start the job. It may take a long time to recover. In some situations, >> stopping jobs is intolerable, for example, the job is related to money or >> important activities.So we need some technologies to control the running >> job without stopping the job. >> >> >> We propose to add control mode for flink. A control mode based on the >> restful interface is first introduced. It works by these steps: >> >> >> 1. The user can predefine some logic which supports config control, >> such as filter condition. >> 2. Run the job. >> 3. If the user wants to change the job's running logic, just send a >> restful request with the responding config. >> >> Other control modes will also be considered in the future. More >> introduction can refer to the doc >> https://docs.google.com/document/d/1WSU3Tw-pSOcblm3vhKFYApzVkb-UQ3kxso8c8jEzIuA/edit?usp=sharing >> . If the community likes the proposal, more discussion is needed and a more >> detailed design will be given later. Any suggestions and ideas are welcome. >> >>
