getOrCreate uses existing SparkSession if available, in which case the
settings will be ignored
On Wed, Nov 16, 2016 at 10:55 PM, bluishpenguin <bluishpengui...@gmail.com>
wrote:
> Hi all,
> I would like to configure the following setting during runtime as below:
>
> spark = (SparkSession
> .builder
> .appName("ElasticSearchIndex")
> .config("spark.kryoserializer.buffer.max", "1g")
> .getOrCreate())
>
> But I still hit error,
> Caused by: org.apache.spark.SparkException: Kryo serialization failed:
> Buffer overflow. Available: 0, required: 1614707. To avoid this, increase
> spark.kryoserializer.buffer.max value.
>
> It works when configure it along with the spark-submit command as below:
> spark-submit pyspark-shell-main --name "PySparkShell" --conf
> spark.kryoserializer.buffer.max=1g
>
> Any idea what have I done wrong?
>
> Thank you.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Configure-spark-kryoserializer-buffer-max-at-
> runtime-does-not-take-effect-tp28094.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
