Thanks all for the answer. Unfortunately while I wasn’t able to use the
extra parameters to get the needed information, I did solve my issue. It
was an issue of using pureconfig to read a certain config from hadoop
before the spark session initialized, therefore pureconfig would error out
in
Hi Ruijing,
Spark uses SerializationDebugger (
https://spark.apache.org/docs/latest/api/java/org/apache/spark/serializer/SerializationDebugger.html)
as default debugger to detect the serialization issues. You can take more
detailed serialization exception information by setting the following
Hi all,
When working with spark jobs, I sometimes have to tackle with serialization
issues, and I have a difficult time trying to fix those. A lot of times,
the serialization issues happen only in cluster mode across the network in
a mesos container, so I can’t debug locally. And the exception
