Hi Andrew, Thanks a lot! Indeed, it doesn't start with spark, the following properties are read by implementation of the driver rather than spark conf:
--conf spooky.root=s3n://spooky- \ --conf spooky.checkpoint=s3://spooky-checkpoint \ This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to set the same properties? Yours Peng On 12 June 2015 at 14:20, Andrew Or <and...@databricks.com> wrote: > Hi Peng, > > Setting properties through --conf should still work in Spark 1.4. From the > warning it looks like the config you are trying to set does not start with > the prefix "spark.". What is the config that you are trying to set? > > -Andrew > > 2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>: > >> In Spark <1.3.x, the system property of the driver can be set by --conf >> option, shared between setting spark properties and system properties. >> >> In Spark 1.4.0 this feature is removed, the driver instead log the >> following >> warning: >> >> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv >> >> How do set driver's system property in 1.4.0? Is there a reason it is >> removed without a deprecation warning? >> >> Thanks a lot for your advices. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
