Thanks all for your information. Andrew, I dig out one of your old post which is relevant:
http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-td5798.html But didn't mention how to supply the properties that don't start with spark. On 12 June 2015 at 19:39, Ted Yu <yuzhih...@gmail.com> wrote: > This is the SPARK JIRA which introduced the warning: > > [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties > in spark-shell and spark-submit > > On Fri, Jun 12, 2015 at 4:34 PM, Peng Cheng <rhw...@gmail.com> wrote: > >> Hi Andrew, >> >> Thanks a lot! Indeed, it doesn't start with spark, the following >> properties are read by implementation of the driver rather than spark conf: >> >> --conf spooky.root=s3n://spooky- \ >> --conf spooky.checkpoint=s3://spooky-checkpoint \ >> >> This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to >> set the same properties? >> >> Yours Peng >> >> On 12 June 2015 at 14:20, Andrew Or <and...@databricks.com> wrote: >> >>> Hi Peng, >>> >>> Setting properties through --conf should still work in Spark 1.4. From >>> the warning it looks like the config you are trying to set does not start >>> with the prefix "spark.". What is the config that you are trying to set? >>> >>> -Andrew >>> >>> 2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>: >>> >>>> In Spark <1.3.x, the system property of the driver can be set by --conf >>>> option, shared between setting spark properties and system properties. >>>> >>>> In Spark 1.4.0 this feature is removed, the driver instead log the >>>> following >>>> warning: >>>> >>>> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv >>>> >>>> How do set driver's system property in 1.4.0? Is there a reason it is >>>> removed without a deprecation warning? >>>> >>>> Thanks a lot for your advices. >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >
