Connectin two jobs with socket over AWS EMR - How to define the ip?

[Itay Sali]

Hi,

I build a streaming pipeline and there are two jobs that I wish to connect with 
socket(later we plan to have kafka instead).
The jobs submitted in AWS EMR cluster with this configuration

    {
      "Classification": "flink-conf",
      "Properties": {
        "JAVA_HOME": "/usr/lib/jvm/java-11-openjdk",
        "env.java.home": "/usr/lib/jvm/java-11-openjdk",
        "high-availability": "zookeeper",
        "high-availability.storageDir": "hdfs:///user/flink/recovery",
        "high-availability.zookeeper.path.root": "/flink",
        "high-availability.zookeeper.quorum": "%{hiera('hadoop::zk')}",
        "java.home": "/usr/lib/jvm/java-11-openjdk",
        "taskmanager.data.port": "35001",
        "taskmanager.numberOfTaskSlots": "2",
        "yarn.application-attempts": "10"
      }
    },


I probably miss how to define the host when I use writeToSocket and 
socketToStream from my jobs.
Is it configuration or one of the primary/core node ips. I have tried many 
options and non of the messages went through.

Any help would be appreciated.

Thanks,
Itay Sali