Assume in my executor I say SparkConf sparkConf = new SparkConf(); sparkConf.set("spark.kryo.registrator", "com.lordjoe.distributed.hydra.HydraKryoSerializer"); sparkConf.set("mysparc.data", "Some user Data"); sparkConf.setAppName("Some App");
Now 1) Are there default values set in some system file which are populated if I call new SparkConf - if not how do I get those? _ I think i see defaults foe the master, the Serializer... 2) If I set a property in SparkConf for my SparkContext will I see that property in a Slave machine? 3) If I set a property anf then call showSparkProperties() do I see that property set and if not how can I see the property set - say in another thread as in if in some other thread on the executor say as in showSparkPropertiesInAnotherThread(); 4) How can a slave machine access properties set on the executor I an really interested in sparkConf.set("spark.kryo.registrator", "com.lordjoe.distributed.hydra.HydraKryoSerializer"); which needs to be used by the Slave /** * dump all spark properties to System.err */ public static void showSparkProperties() { SparkConf sparkConf = new SparkConf(); Tuple2<String, String>[] all = sparkConf.getAll(); for (Tuple2<String, String> prp : all) { System.err.println(prp._1().toString() + "=" + prp._2()); } } public static void showSparkPropertiesInAnotherThread() { new Thread(new Runnable() { @Override public void run() { showSparkProperties(); } }).start(); }