Hi all, I tried to set the configuration spark.sql.inMemoryColumnarStorage.compressed, and spark.sql.inMemoryColumnarStorage.batchSize in spark.executor.extraJavaOptions but it does not work, my spark.executor.extraJavaOptions contains "Dspark.sql.inMemoryColumnarStorage.compressed=true -Dspark.sql.inMemoryColumnarStorage.batchSize=1000000", and SparkUI did indicate that the same setting, but somehow the memory footprint of my cacheTable RDDs are the same as without the setting. Only when I use HiveContext.setConf, it reduce my RDDs memory usage. Is it a bug here, or user are required to set the config using HiveContext's setConf function?
Regards, Kelvin Paul
