What algorithm? Can you provide code? On Fri, Feb 12, 2016 at 3:22 PM, Raghava Mutharaju < m.vijayaragh...@gmail.com> wrote:
> Hello All, > > I implemented an algorithm using both the RDDs and the Dataset API (in > Spark 1.6). Dataset version takes lot more memory than the RDDs. Is this > normal? Even for very small input data, it is running out of memory and I > get a java heap exception. > > I tried the Kryo serializer by registering the classes and I > set spark.kryo.registrationRequired to true. I get the following exception > > com.esotericsoftware.kryo.KryoException: > java.lang.IllegalArgumentException: Class is not registered: > org.apache.spark.sql.types.StructField[] > Note: To register this class use: > kryo.register(org.apache.spark.sql.types.StructField[].class); > > I tried registering > using conf.registerKryoClasses(Array(classOf[StructField[]])) > > But StructField[] does not exist. Is there any other way to register it? I > already registered StructField. > > Regards, > Raghava. >
