Hello,
<https://stackoverflow.com/posts/76378281/timeline>
I can launch a Flink cluster (version 1.17.x) on my laptop with 1 Job
Manager and 3 Task Managers. The cluster starts, jobs can be submitted
correctly on the localhost (my laptop).
Next I tried to launch this cluster on 4 VMs - 1 Master VM (for the Job
Manager) and 3 Worker VMs (for Task Managers). I am not using YARN, K8s or
Docker on any of these VMs.
The cluster starts up fine using "${FLINK_HOME}"/bin/start-cluster.sh.
i.e., running this on the command line on the Master VM does the expected -
it starts the job manager on the Master and then starts 1 Task Manager on
each Worker VM. (ssh connectivity between Master and Worker VMs is fine).
However, job submission("${FLINK_HOME}"/bin/flink run myapp.jar) fails with
the following exceptions:
> Caused by: org.apache.flink.util.FlinkException: Could not upload job files.
> at
> org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:86)
> at
> org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$uploadJobGraphFiles$4(JobSubmitHandler.java:195)
> ... 10 more
> Caused by: java.io.IOException: Could not connect to BlobServer at address
> localhost/127.0.0.1:37452
> at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:103)
> at
> org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$null$3(JobSubmitHandler.java:199)
> at
> org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:82)
> ... 11 more
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
> at
> java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
> at
> java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.base/java.net.Socket.connect(Socket.java:615)
> at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:97)
> ... 13 more
> ]
> at
> org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:536)
> at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:516)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
> ... 4 more
>
>
Looking at the DEBUG level logs on the Master (Job Manager) and the Workers
(Task Managers) ... it looks like the Task Managers on the Worker VMS are
all registered with the Job Manager on the Master VM (The Job Manager logs
shows the Task Managers). Hence, connectivity and registration from Worker
VMs back to the Master VM (and the job manager) is fine.
The Web UI reflects this too. I can see the Job Manager and 3 Task managers
running on those 4 VMs.
Hence, it is ONLY the job submission that's failing with this "Could not
connect to Blob Server" exception. I suspect I have the configuration
incorrect.
Questions:
1. Is there a reference implementation of a Multi-VM Flink Cluster (NOT
on Docker)?
2. How should taskmanager.*.*.* properties in flink-conf.yaml be
configured for a multi-VM cluster? In particular, what should the values of
"taskmanager.bind-host" and "task-manager.host" be on the Master VM?
3. Does a multi-VM flink cluster need a shared storage directory? If so,
is there any documentation on configuring this?
4. From reading various Flink enhancement requests and mailing list
posts, I found that the Blob Server needs to be accessible from the Task
Managers on an address that's external facing. Note from the exception
trace above that the attempt to connect to the BlobServer is being made on
"localhost/127.0.0.1:37452". I cannot find any configuration item that
allows me to set the host or bind-host for the blobserver. How is this to
be done?
Please advise. Thanks in advance for your responses.
