Hi It could also be conf/spark-defaults.conf.
Jacek 11.03.2016 8:07 PM "Cesar Flores" <ces...@gmail.com> napisał(a): > > Right now I know of three different things to pass property parameters to > the Spark Context. They are: > > - A) Inside a SparkConf object just before creating the Spark Context > - B) During job submission (i.e. --conf spark.driver.memory.memory = > 2g) > - C) By using a specific property file during job submission (i.e. > --properties-file somefile.conf) > > My question is: If you specify the same config parameter in more than one > place *what is the one that will be actually used?* > > *Does the priority order is the same for any property or is property > dependent?* > > *I am mostly interested in the config parameter > spark.sql.shuffle.partitions, which I need to modify on the fly to do group > by clauses depending on the size of my input.* > > > Thanks > -- > Cesar Flores >
