Questions about serialization and SparkConf

2014-10-29 Thread Steve Lewis
Assume in my executor I say SparkConf sparkConf = new SparkConf(); sparkConf.set(spark.kryo.registrator, com.lordjoe.distributed.hydra.HydraKryoSerializer); sparkConf.set(mysparc.data, Some user Data); sparkConf.setAppName(Some App); Now 1) Are there default

Re: Questions about serialization and SparkConf

2014-10-29 Thread Ilya Ganelin
Hello Steve . 1) When you call new SparkConf you should get an object with the default config values. You can reference the spark configuration and tuning pages for details on what those are. 2) Yes. Properties set in this configuration will be pushed down to worker nodes actually executing the