Hi All,
I try to pass parameter to the spark-shell when I do some test: spark-shell --driver-memory 512M --executor-memory 4G --master spark://****:7077 --conf spark.sql.parquet.compression.codec=snappy --conf spark.sql.parquet.binaryAsString=true This works fine on my local pc. And when I start EMR, and pass the similar things: ~/spark/bin/spark-shell --executor-memory 40G --master yarn-client --num-executors 1 --executor-cores 32 --conf spark.sql.parquet.compression.codec=snappy --conf spark.sql.parquet.binaryAsString=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer The parameter doesn't pass to spark-shell. Anyone knows why? And what is the alternatives for me to do that? Regards, Shuai