Hi, If it is for simplicity, you can also try writing the flag into an external system, such as Redis、Zookeeper or MySQL, and query the flag from the external system when perform data processing.
However, Broadcast State is still the mode that I recommend. Perhaps we only need to encapsulate the repetitive logic (reading data from a topic) by defining a ConfigSource, for example, to handle reading data from a topic and converting it into the interested configuration items, so that independent operators can be reused that logic. Additionally, I have attached some articles about the usage of Broadcast State[1][2]. Hope it helps. Best, Jiabao [1] https://flink.apache.org/2019/06/26/a-practical-guide-to-broadcast-state-in-apache-flink/ [2] https://flink.apache.org/2020/03/24/advanced-flink-application-patterns-vol.2-dynamic-updates-of-application-logic/ On 2023/12/07 16:18:42 Oscar Perez via user wrote: > Hi, > We would like to enable sort of a feature flag functionality for flink jobs. > > The idea would be to use broadcast state reading from a configuration topic > and then ALL operators with logic would listen to this state. > > This documentation: > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/fault-tolerance/broadcast_state/ > > explains how a certain operator can use this broadcast state but the > problem we are having is understanding how we can share the same state > across many different operators. One way is to create multiple streams, one > per operator reading from the same topic and then connect to the multiple > operators in a keyedbroadcastprocessfunction but this seems overkill > > Is there an elegant solution to this problem? > regards, > Oscar >
