Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Timo, Sure, I have opened this issue on Jira. On Fri, Oct 23, 2020 at 4:09 PM Timo Walther wrote: > Hi Manas, > > that is a good point. Feel free to open an issue for this. It is not the > first time that your question appeared on the mailin

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Manas, that is a good point. Feel free to open an issue for this. It is not the first time that your question appeared on the mailing list. Regards, Timo On 23.10.20 07:22, Manas Kale wrote: Hi Timo, I figured it out, thanks a lot for your help. Are there any articles detailing the pre-fl

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Timo, I figured it out, thanks a lot for your help. Are there any articles detailing the pre-flight and cluster phases? I couldn't find anything on ci.apache.org/projects/flink and I think this behaviour should be documented as a warning/note. On Thu, Oct 22, 2020 at 6:44 PM Timo Walther wrot

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Manas, you can use static variable but you need to make sure that the logic to fill the static variable is accessible and executed in all JVMs. I assume `pipeline.properties` is in your JAR that you submit to the cluster right? Then you should be able to access it through a singleton patt

Re: Flink - Kafka topic null error; happens only when running on cluster

Sorry, I messed up the code snippet in the earlier mail. The correct one is : public static void main(String[] args) { Properties prop = new Properties(); InputStream is = Config.class.getClassLoader().getResourceAsStream("pipeline.properties"); prop.load(is); HashMap strMap = new HashMap

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Timo, Thank you for the explanation, I can start to see why I was getting an exception. Are you saying that I cannot use static variables at all when trying to deploy to a cluster? I would like the variables to remain static and not be instance-bound as they are accessed from multiple classes. B

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Manas, you need to make sure to differentiate between what Flink calls "pre-flight phase" and "cluster phase". The pre-flight phase is were the pipeline is constructed and all functions are instantiated. They are then later serialized and send to the cluster. If you are reading your pro

Flink - Kafka topic null error; happens only when running on cluster

Hi, I am trying to write some data to a kafka topic and I have the following situation: monitorStateStream .process(new IDAP2AlarmEmitter()).name(IDAP2_ALARM_EMITTER).uid(IDAP2_ALARM_EMITTER) *... // Stream that outputs elements of type IDAP2Alarm* .addSink(getFlinkKafkaProducer(ALARMS_KA