Hi, I think this would be the very basic use case for Broadcast State Pattern but I would like to know what are the best approaches to solve this problem.
I have an operator that extends BroadcastProcessFunction. The brodcastElement is an element sent as Json format message by Kafka. It describes a processing rules like key/value mapping, like so: ruleName - ruleValue (both strings). In processElement method I'm delegating to my custom RuleEngineService. It is a class that has the "rule engine" logic and accepts received event and "set of processing rules" in some form. What would be the best approaches: 1. Keep original Json String in broadcast state. Whenever there is a new set of rules streamed by Kafka, then in processBroadcastElement method parse this Json, map to some RuleParams abstraction and keep it as transient field in my BroadcastProcessFunction operator. Save Json in broadcast state. Pass RuleParams to rule engine service. 2. Same as 1 but instead keeping Raw Json String in broadcast state, keep already parsed JsonObject, somethign like ObjectNode from KafkaConnector lib. 3. Keep each pair of ruleName - ruleValue (both strings) separate in broadcast state. In processBrodcastElement method parse the received Json and update the state. In processElement method take all rules, build RulePArams object (basically a map) and pass them to rule engine 4. Parse Json in processBroadcastElement method, map it to RuleParams abstraction method, keeping rules in a hashMap and keep this RulePrams in broadcast state 5. any other... -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
