Hi Andrew,
Thanks a lot! Indeed, it doesn't start with spark, the following properties
are read by implementation of the driver rather than spark conf:
--conf spooky.root=s3n://spooky- \
--conf spooky.checkpoint=s3://spooky-checkpoint \
This used to work from Spark 1.0.0 to 1.3.1. Do you know
This is the SPARK JIRA which introduced the warning:
[SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties
in spark-shell and spark-submit
On Fri, Jun 12, 2015 at 4:34 PM, Peng Cheng rhw...@gmail.com wrote:
Hi Andrew,
Thanks a lot! Indeed, it doesn't start with spark, the
Thanks all for your information. Andrew, I dig out one of your old post
which is relevant:
http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-td5798.html
But didn't mention how to supply the properties that don't start with spark.
On 12 June
In Spark 1.3.x, the system property of the driver can be set by --conf
option, shared between setting spark properties and system properties.
In Spark 1.4.0 this feature is removed, the driver instead log the following
warning:
Warning: Ignoring non-spark config property: xxx.xxx=v
How do
Hi Peng,
Setting properties through --conf should still work in Spark 1.4. From the
warning it looks like the config you are trying to set does not start with
the prefix spark.. What is the config that you are trying to set?
-Andrew
2015-06-12 11:17 GMT-07:00 Peng Cheng pc...@uow.edu.au:
In
