On Thu, Mar 26, 2020, 8:35 PM Andrew Melo <andrew.m...@gmail.com> wrote:
> Hello all,
>
> Is there a way to register classes within a datasourcev2 implementation in
> the Kryo serializer?
>
> I've attempted the following in both the constructor and static block of
> my toplevel class:
>
> SparkContext context = SparkContext.getOrCreate();
> SparkConf conf = context.getConf();
> Class[] classesRegistered = new Class[] {
> edu.vanderbilt.accre.laurelin.spark_ttree.Reader.class,
> edu.vanderbilt.accre.laurelin.spark_ttree.Partition.class,
> edu.vanderbilt.accre.laurelin.spark_ttree.SlimTBranch.class
> };
> conf.registerKryoClasses(classesRegistered);
>
> But (if I'm reading correctly), this is too late, since the config has
> already been parsed while initializing the SparkContext, adding classes to
> the SparkConf has no effect. From what I can tell, the kryo instance behind
> is private, so I can't add the registration manually either.
>
> Any thoughts?
> Thanks
> Andrew
>
>
>
