Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
Here he's referring to a line of code that calls SparkConf.getenv vs System.getenv, but the former calls the latter. In neither case does it read from a props file. On Tue, Feb 16, 2016 at 10:44 PM, Igor Costa wrote: > Actually answering the first question: > > Is there a reason to use conf to read SPARK_WORKER_MEMORY not > System.getenv as for the other env vars? > > You can use the properties file to change the amount, System.getenv would be > bad when you have for example other things running on the JVM which will > cause conflict on some parts. > Defined usage in properties files is more convenience for custom UI to be > made available. > > On Sat, Feb 13, 2016 at 8:38 PM, Sean Owen wrote: >> >> Yes you said it is only set in a props file, but why do you say that? >> because the resolution of your first question is that this is not >> differently handled. >> >> On Fri, Feb 12, 2016 at 11:11 PM, Jacek Laskowski wrote: >> > On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen wrote: >> >> I think that difference in the code is just an oversight. They >> >> actually do the same thing. >> > >> > Correct. Just meant to know the reason if there was any. >> > >> >> Why do you say this property can only be set in a file? >> > >> > I said that conf/spark-defaults.conf can *not* be used to set >> > spark.worker.ui.port property and wondered why is so? It'd be nice to >> > have it for settings (not use workarounds like >> > SPARK_WORKER_OPTS=-Dspark.worker.ui.port=21212). Just spot it and >> > thought I'd ask if it needs to be cleaned up or improved. >> > >> > Jacek >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
Actually answering the first question: Is there a reason to use conf to read SPARK_WORKER_MEMORY not System.getenv as for the other env vars? You can use the properties file to change the amount, System.getenv would be bad when you have for example other things running on the JVM which will cause conflict on some parts. Defined usage in properties files is more convenience for custom UI to be made available. On Sat, Feb 13, 2016 at 8:38 PM, Sean Owen wrote: > Yes you said it is only set in a props file, but why do you say that? > because the resolution of your first question is that this is not > differently handled. > > On Fri, Feb 12, 2016 at 11:11 PM, Jacek Laskowski wrote: > > On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen wrote: > >> I think that difference in the code is just an oversight. They > >> actually do the same thing. > > > > Correct. Just meant to know the reason if there was any. > > > >> Why do you say this property can only be set in a file? > > > > I said that conf/spark-defaults.conf can *not* be used to set > > spark.worker.ui.port property and wondered why is so? It'd be nice to > > have it for settings (not use workarounds like > > SPARK_WORKER_OPTS=-Dspark.worker.ui.port=21212). Just spot it and > > thought I'd ask if it needs to be cleaned up or improved. > > > > Jacek > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
Yes you said it is only set in a props file, but why do you say that? because the resolution of your first question is that this is not differently handled. On Fri, Feb 12, 2016 at 11:11 PM, Jacek Laskowski wrote: > On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen wrote: >> I think that difference in the code is just an oversight. They >> actually do the same thing. > > Correct. Just meant to know the reason if there was any. > >> Why do you say this property can only be set in a file? > > I said that conf/spark-defaults.conf can *not* be used to set > spark.worker.ui.port property and wondered why is so? It'd be nice to > have it for settings (not use workarounds like > SPARK_WORKER_OPTS=-Dspark.worker.ui.port=21212). Just spot it and > thought I'd ask if it needs to be cleaned up or improved. > > Jacek - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen wrote: > I think that difference in the code is just an oversight. They > actually do the same thing. Correct. Just meant to know the reason if there was any. > Why do you say this property can only be set in a file? I said that conf/spark-defaults.conf can *not* be used to set spark.worker.ui.port property and wondered why is so? It'd be nice to have it for settings (not use workarounds like SPARK_WORKER_OPTS=-Dspark.worker.ui.port=21212). Just spot it and thought I'd ask if it needs to be cleaned up or improved. Jacek - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
I think that difference in the code is just an oversight. They actually do the same thing. Why do you say this property can only be set in a file? On Fri, Feb 12, 2016 at 9:39 PM, Jacek Laskowski wrote: > Hi devs, > > Following up on this, it appears that spark.worker.ui.port can only be > set in --properties-file. I wonder why conf/spark-defaults.conf is > *not* used for the spark.worker.ui.port property? Any reason for the > decision? > > Pozdrawiam, > Jacek > > Jacek Laskowski | https://medium.com/@jaceklaskowski/ > Mastering Apache Spark > ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ > Follow me at https://twitter.com/jaceklaskowski > > > On Thu, Feb 11, 2016 at 2:51 PM, Jacek Laskowski wrote: >> Hi, >> >> Is there a reason to use conf to read SPARK_WORKER_MEMORY not >> System.getenv as for the other env vars? >> >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala#L45 >> >> Pozdrawiam, >> Jacek >> >> Jacek Laskowski | https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark >> ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ >> Follow me at https://twitter.com/jaceklaskowski > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
Hi devs, Following up on this, it appears that spark.worker.ui.port can only be set in --properties-file. I wonder why conf/spark-defaults.conf is *not* used for the spark.worker.ui.port property? Any reason for the decision? Pozdrawiam, Jacek Jacek Laskowski | https://medium.com/@jaceklaskowski/ Mastering Apache Spark ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ Follow me at https://twitter.com/jaceklaskowski On Thu, Feb 11, 2016 at 2:51 PM, Jacek Laskowski wrote: > Hi, > > Is there a reason to use conf to read SPARK_WORKER_MEMORY not > System.getenv as for the other env vars? > > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala#L45 > > Pozdrawiam, > Jacek > > Jacek Laskowski | https://medium.com/@jaceklaskowski/ > Mastering Apache Spark > ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ > Follow me at https://twitter.com/jaceklaskowski - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?
Hi, Is there a reason to use conf to read SPARK_WORKER_MEMORY not System.getenv as for the other env vars? https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala#L45 Pozdrawiam, Jacek Jacek Laskowski | https://medium.com/@jaceklaskowski/ Mastering Apache Spark ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ Follow me at https://twitter.com/jaceklaskowski - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
