When the FlinkRunner (python client) sees flink_master as [auto] or [local], then it could set the default environment to LOOPBACK before the pipeline is constructed and provide the loopback environment. Isn't that fully client side controlled?
There is the case of a pre-configured job server with a master url. In this case we probably do not want LOOPBACK execution because the job server will submit the pipeline to a an actual cluster. If we make the assumption that we do not allow that for the Python FlinkRunner, setting LOOPBACK in this case seems fair.
On 31.10.19 16:04, Thomas Weise wrote:
On Thu, Oct 31, 2019 at 3:55 AM Maximilian Michels <m...@apache.org <mailto:m...@apache.org>> wrote:> Thanks for clarifying. So when I run "./flink my_pipeline.jar" or > upload the jar via the REST API (and its main method invoked on the > master) then [auto] reads the config and does the right thing, but if > I do java my_pipeline.jar it'll run locally. Correct. > Python needs to know even whether to start up the loopback server, and > provide the address when submitting the pipeline. I was thinking, it could do this anyway and tear down that server if the Runner does not need it. Clearly not the ideal solution.When the FlinkRunner (python client) sees flink_master as [auto] or [local], then it could set the default environment to LOOPBACK before the pipeline is constructed and provide the loopback environment. Isn't that fully client side controlled?
