Re: Running on a firewalled Yarn cluster?

I've finally fixed the issues identified here in the thread: The blob manager and the application master/job manager allocate their ports in a specified range. You can now whitelist a port range in the firewall and Flink services will only allocate ports in that range:

Re: Running on a firewalled Yarn cluster?

Hi, I just wanted to let you know that I didn't forget about this! The BlobManager in 1.0-SNAPSHOT has already a configuration parameter to use a certain range of ports. I'm trying to add the same feature for YARN tomorrow. Sorry for the delay. On Tue, Nov 10, 2015 at 9:27 PM, Cory Monty

Re: Running on a firewalled Yarn cluster?

While discussing with my colleagues about the issue today, we came up with another approach to resolve the issue: d) Upload the job jar to HDFS (or another FS) and trigger the execution of the jar using an HTTP request to the web interface. We could add some tooling into the /bin/flink client to

Re: Running on a firewalled Yarn cluster?

Hi, I checked and this setting has been set to a limited port range of only 100 port numbers. I tried to find the actual port an AM is running on and couldn't find it (I'm not the admin on that cluster) The url to the AM that I use to access it always looks like this:

Re: Running on a firewalled Yarn cluster?

Great! I'll watch the issue and give it a test once I see a working patch. Niels Basjes On Tue, Nov 3, 2015 at 1:03 PM, Maximilian Michels wrote: > Hi Niels, > > Thanks a lot for reporting this issue. I think it is a very common setup > in corporate infrastructure to have

Re: Running on a firewalled Yarn cluster?

Hi, I forgot to answer your other question: On Mon, Nov 2, 2015 at 4:34 PM, Robert Metzger wrote: > so the problem is that you can not submit a job to Flink using the > "/bin/flink" tool, right? > I assume Flink and its TaskManagers properly start and connect to each >

Re: Running on a firewalled Yarn cluster?

Hi Niels, so the problem is that you can not submit a job to Flink using the "/bin/flink" tool, right? I assume Flink and its TaskManagers properly start and connect to each other (the number of TaskManagers is shown correctly in the web interface). I see the following solutions for the problem

Running on a firewalled Yarn cluster?

Hi, Here at work our security guys have chosen (long time ago) to only allow the firewalls to have the ports open that needed (I say: good call!). For the Yarn cluster this includes things like the proxy to see the application manager of an application. For everything we've done so far (i.e.

Re: Running on a firewalled Yarn cluster?

My take on those 3 options: a) Bad idea; people need to be able to automate their jobs and run them from the command line (i.e. bash, cron). b) Bad idea; Same reason you gave. In addition I do not want to reserve an open 'flink port' for every user who wants to run a job. c) From my perspective