[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864595#comment-15864595
]
Julian Gamble commented on SPARK-9279:
--
This is a serious issue in Cloud/Docker environments when trying to look at
Spark workers through a proxy (like Fabio) that defaults to 80. You need the
worker to publish its port on 80 (regardless of what port the docker host
opens) so that its published port is 80 for the proxy. (You can work around
this with Ambassadors - but other applications don't force you to do this.)
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1977)
> at
> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
> at org.apache.spark.
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610642#comment-15610642
]
Suresh commented on SPARK-9279:
---
Folks this limitation to not allow binding Spark UI to port 80, even as root is
serious issue. Here is my use case and I am sure lot of folks will hit this
issue.
I have a Spark running in AWS and it is for my personal use case of learning.
Now whatever I do I cannot access it's Web UI when I am in my office.
Reason - Our office corporate network has blocked http access to any external
site which does not listen on 80 OR 443. This is not very strange policy and is
used by many companies.
Now, how do I learn about Spark and experiment with it ? Don't do it while in
office. ?
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649603#comment-14649603
]
Sean Owen commented on SPARK-9279:
--
No, that's not the issue, even if I don't think it's great to run on port 80.
It's the retry logic. You have to globally disable the sequential port retry
logic or else this won't make sense for privileged ports. I think that this is
a subset of http://issues.apache.org/jira/browse/SPARK-4449 : if you can
specify ports or ranges separately for every port, then there's no reason to
restrict privileged ports. That JIRA / PR contains a lot of discussion about
why that is also a little tricky to implement. It's not insurmountable. It's
dealing with the potential API/behavior changes as carefully as possible. Have
a read of that and see what you think.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649570#comment-14649570
]
Daniel Li commented on SPARK-9279:
--
I echo Omar's point - "I understand that my particular use goes against normal
best practices, but I don't think it's a developer's place to enforce these
kinds of constraints on users just because it makes sense to them. It seems to
limit spark's usability for no gain elsewhere"
I have a case where users can access services running in data center only
through a small number ports between 1 and 1024. Hard-code spark webui port
essentially stopped users from access spark webui in the data center.
I submitted a pull request above (https://github.com/apache/spark/pull/7835).
Sean, do you have major concerns in configuring spark webui to use ports
between 1 and 1024?
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649532#comment-14649532
]
Apache Spark commented on SPARK-9279:
-
User 'danielli90' has created a pull request for this issue:
https://github.com/apache/spark/pull/7835
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1977)
> at
> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
> at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
> at org.apache.spark.deploy.master.Master.preStart(Master.scala:144)
> at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
> at
> org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52)
> at akka.a
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639331#comment-14639331
]
Sean Owen commented on SPARK-9279:
--
No it's the per-app driver, not the master, but the code path is the same. I'm
trying to work out why this check is there in the first place; there's a reason
that may not matter much for the master but does elsewhere. (The retry doesn't
make as much sense for the master, which makes me think I might be missing
something about why it is even used for the master.) I think it's at least
possible to handle the 0-retries case differently; that's not invasive.
Disabling retry on the master port is a slightly bigger behavior change (that
would mean there's no reason to prohibit well-known ports too) but it's
possible too.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
>
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639298#comment-14639298
]
Omar Padron commented on SPARK-9279:
Forgive me for not being fully aware of the various terminology. Is the
"driver" in this case the spark master server?
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1977)
> at
> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
> at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
> at org.apache.spark.deploy.master.Master.preStart(Master.scala:144)
> at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
> at
> org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52)
>
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639284#comment-14639284
]
Sean Owen commented on SPARK-9279:
--
The default is to create a web UI for the driver, and without retry, any time
two drivers were run on the same machine (think of a shared cluster) one would
fail. It would be up to the user to guess another port and try again. As I say,
it's (globally) configurable, for cases where it's important to fix the ports,
though being global is still a disadvantage. Would you like to make a PR to
allow any port if the # of retries is 0? That enables this case but remains
backwards compatible.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1977)
> at
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639273#comment-14639273
]
Omar Padron commented on SPARK-9279:
For me, trying a bunch of ports in sequence strikes me as poorer practice than
trying to run a webui on port 80. If a user provides a port to run on, it
should run on that port or fail (for reasons other than "I don't like this
port.").
I would recommend some kind of option for invoking this port-scanning behavior
and to change the default behavior to something more sane.
As far as my particular use case goes, it would be sufficient to simply honor
the user-provided port, even if it is a privileged port.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Util
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639253#comment-14639253
]
Sean Owen commented on SPARK-9279:
--
On looking at this code again, I think the issue is that this method always
tries to bind to a number of ports in sequence until one is available. this
won't make sense for privileged / well known ports, which is why it's precluded
I believe.
There's a way to disable this retry with spark.port.retries=0. It's not a great
system since this is global, and I know there _was_ a JIRA around here
somewhere for implementing more elaborate port syntax to specify ranges, but
don't think it went anywhere.
As a small step forward, I think the logic here could allow a privileged port
if spark.port.retries=0 since there's no problem with trying a sequence of
ports. That's probably the behavior that you want in this scenario anyway.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirem
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639242#comment-14639242
]
Omar Padron commented on SPARK-9279:
No, it is not *absolutely* necessary for me (hence, the minor priority).
I understand that my particular use goes against normal best practices, but I
don't think its a developer's place to enforce these kinds of constraints on
users just because it makes sense to them. It seems to limit spark's usability
for no gain elsewhere.
For another data point, I'd point out that I've worked with the WebUI's in HDFS
as well as YARN in similar configurations without issue. So, if you'd have
software dictate these kinds of constraints, you should at least do so
consistently across projects, imo.
Thank you for your consideration.
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639213#comment-14639213
]
Sean Owen commented on SPARK-9279:
--
Oh I see, I apologize, I read right past that at the start. It's probably not a
great idea to run it as root just to bind to port 80. Being one of several app
UIs, I am not sure one Spark master on a machine needs to use the well known
HTTP port. That's why the limit exists. I also don't see a reason this must be
explicitly forbidden, but do you really need to do this?
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.MasterWebUI (Logging.scala:logError(96)) - Failed to bind MasterWebUI
> java.lang.IllegalArgumentException: requirement failed: startPort should be
> between 1024 and 65535 (inclusive), or 0 for a random free port.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1977)
> at
> org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
> at org.apache.spark.ui.WebUI.bind(WebUI
[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port
[
https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639150#comment-14639150
]
Omar Padron commented on SPARK-9279:
bq. This has nothing to do with Spark. Any Linux-like OS X requires root
privileges for any process to bind to a port under 1024.
Am I missing something?
bq. When trying to start a spark master server as root...
bq. ... Users choosing to run spark as root should be allowed to choose their
own ports.
I've mentioned twice that I am running spark as root, including on the very
first sentence of the bug description.
In case I was not clear: I am reporting on an issue with the port selection
logic that is *explicitly coded* into spark. This is not a matter of not
having access to the privileged ports, but of spark explicitly refusing to bind
to them. This has everything to do with spark.
See:
https://github.com/apache/spark/blob/c032b0bf92130dc4facb003f0deaeb1228aefded/core/src/main/scala/org/apache/spark/util/Utils.scala#L2019
> Spark Master Refuses to Bind WebUI to a Privileged Port
> ---
>
> Key: SPARK-9279
> URL: https://issues.apache.org/jira/browse/SPARK-9279
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
> Environment: Ubuntu Trusty running in a docker container
>Reporter: Omar Padron
>Priority: Minor
>
> When trying to start a spark master server as root...
> {code}
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=80
> spark-class org.apache.spark.deploy.master.Master \
> --host "$( hostname )" \
> --port "$SPARK_MASTER_PORT" \
> --webui-port "$SPARK_MASTER_WEBUI_PORT"
> {code}
> The process terminates with IllegalArgumentException "requirement failed:
> startPort should be between 1024 and 65535 (inclusive), or 0 for a random
> free port."
> But, when SPARK_MASTER_WEBUI_PORT=8080 (or anything >1024), the process runs
> fine.
> I do not understand why the usable ports have been arbitrarily restricted to
> the non-privileged. Users choosing to run spark as root should be allowed to
> choose their own ports.
> Full output from a sample run below:
> {code}
> 2015-07-23 14:36:50,892 INFO [main] master.Master
> (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM,
> HUP, INT]
> 2015-07-23 14:36:51,399 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2015-07-23 14:36:51,586 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing view acls to: root
> 2015-07-23 14:36:51,587 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - Changing modify acls to: root
> 2015-07-23 14:36:51,588 INFO [main] spark.SecurityManager
> (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui
> acls disabled; users with view permissions: Set(root); users with modify
> permissions: Set(root)
> 2015-07-23 14:36:52,295 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
> 2015-07-23 14:36:52,349 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
> 2015-07-23 14:36:52,489 INFO [sparkMaster-akka.actor.default-dispatcher-2]
> Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening
> on addresses :[akka.tcp://sparkMaster@sparkmaster:7077]
> 2015-07-23 14:36:52,497 INFO [main] util.Utils (Logging.scala:logInfo(59)) -
> Successfully started service 'sparkMaster' on port 7077.
> 2015-07-23 14:36:52,717 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started
> SelectChannelConnector@sparkmaster:6066
> 2015-07-23 14:36:52,759 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> util.Utils (Logging.scala:logInfo(59)) - Successfully started service on port
> 6066.
> 2015-07-23 14:36:52,760 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> rest.StandaloneRestServer (Logging.scala:logInfo(59)) - Started REST server
> for submitting applications on port 6066
> 2015-07-23 14:36:52,765 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Starting Spark master at
> spark://sparkmaster:7077
> 2015-07-23 14:36:52,766 INFO [sparkMaster-akka.actor.default-dispatcher-4]
> master.Master (Logging.scala:logInfo(59)) - Running Spark version 1.4.1
> 2015-07-23 14:36:52,772 ERROR [sparkMaster-akka.actor.default-dispatcher-4]
> ui.Maste
