Docker Flink Official Image with Docker Swarm cannot register TaskManager to JobManager's Resource Manager

[chyp long]

Hello, I am having trouble when deploying Apache Flink 1.16.1 on 2 Google
Cloud instances with Docker Swarm. The JobManager is deployed on the
manager node and the TaskManager is deployed in the worker node. The
TaskManager seems to have trouble to communicate with ResourceManager on
JobManager via akka.tcp. Here is the log on the TaskManager

flink_taskmanager.1.e7sxy43ls...@workernode.gcp    | 2023-03-29
13:50:34,061 INFO
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] -
Start job leader service.
flink_taskmanager.1.e7sxy43ls...@workernode.gcp    | 2023-03-29
13:50:34,066 INFO  org.apache.flink.runtime.filecache.FileCache
         [] - User file cache uses directory
/tmp/flink-dist-cache-705fdca5-c285-4875-9b68-556ccd1b56c3
flink_taskmanager.1.e7sxy43ls...@workernode.gcp    | 2023-03-29
13:50:34,073 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
         [] - Connecting to ResourceManager
akka.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000).
flink_taskmanager.1.e7sxy43ls...@workernode.gcp    | 2023-03-29
13:50:34,420 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
         [] - Resolved ResourceManager address, beginning registration
flink_taskmanager.1.e7sxy43ls...@workernode.gcp    | 2023-03-29
13:55:34,086 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor
         [] - Fatal error occurred in TaskExecutor
akka.tcp://flink@10.0.1.14:6127/user/rpc/taskmanager_0.
flink_taskmanager.1.e7sxy43ls...@workernode.gcp    |
org.apache.flink.runtime.taskexecutor.exceptions.RegistrationTimeoutException:
Could not register at the ResourceManager within the specified maximum
registration duration PT5M. This indicates a problem with this
instance. Terminating now.

About firewall rules, these instances are deployed in the same VPC,
same subnet, so I suppose it can communicate without any troubles. In
fact, I can ping or curl on both instances. Before deploying to Google
Cloud, I have successfully deployed the same setup on my private
Microstack cloud without any problem. Here is my docker-compose file
that I used docker stack deploy:

version: '3.8'

services:
  jobmanager:
    image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
    deploy:
      replicas: 1
      placement:
        constraints: [node.hostname == managernode.gcp]
    ports:
      - "8081:8081"
      - "6123:6123"
      - "6124:6124"
      - "6125:6125"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=${FLINK_PROPERTIES}
    networks:
      - flink-network
  taskmanager:
    image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
    deploy:
      replicas: 1
      placement:
        constraints: [node.hostname == workernode.gcp]
    depends_on:
      - jobmanager
    ports:
      - "6121:6121"
      - "6122:6122"
      - "6126:6126"
      - "6127:6127"
      - "6128:6128"
      - "5005:5005/udp"
    command:
      - taskmanager
    environment:
      - FLINK_PROPERTIES=${FLINK_PROPERTIES}
    networks:
      - flink-network

networks:
  flink-network:
    driver: overlay
    attachable: true

FLINK_PROPERTIES

FLINK_PROPERTIES=$'\njobmanager.rpc.address:
jobmanager\nparallelism.default: 2\n'

I am using a customized flink docker image to fix taskmanager.data.port and
taskmanager.rpc.port to 6126 and 6127. I have tried to change
jobmanager.rpc.address with the private IP and zonal DNS, ResourceManager
can register the taskmanager. However, by doing so, flink-metrics is unable
to work. I expect that I can successfully deploy flink cluster on Google
Cloud instances with Docker Swarm, same with what I did on Microstack.