Hello folks, I'm trying to deploy a Spark driver on Amazon EMR in yarn-cluster mode expecting to be able to access the Spark UI from the <spark-master-ip>:4040 address (default port). The problem here is that the Spark UI port is always defined randomly at runtime, although I also tried to specify it in the spark-defaults.conf file: in order to do so, I used this: https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark#3-utilize-an-emr-step-to-configure-the-spark-default-configuration-optional , setting the spark.ui.port to a static known value. No luck, every time I launch (using the spark-submit script from the yarn-master node) a Spark driver, the UI port is chose randomly.
Is there any configurations I'm missing out here? Thank you very much. Roberto
