Looks like what I am currently doing, or at least close.No need to copy the big file on every node.Copy on one node. Read the data, and send it to a Kafka cluster using KafkaProducer() object.Use KafkaIO() (in case its a Beam app). Deploy to the node where the JM is running.It will be executed in s distributed fashion across all nodes.If that help, I can privately help you how the logistics & the code may look like + a loooooooooooot of tricksI have learned the hard way LOL!Thanks. Amir-
From: Mariano Gonzalez <mariano.gonza...@uptake.com> To: dev@flink.apache.org Sent: Thursday, September 29, 2016 3:07 PM Subject: Re: Use specific Task Manager for heavy computations Any ideas? -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Use-specific-Task-Manager-for-heavy-computations-tp13747p13771.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.