HI Shuai,
I don't think this is a bug with kryo, its just a subtlety with the kryo
works. I *think* that it would also work if you changed your
PropertiesUtil class to either (a) remove the no-arg constructor or (b)
instead of extending properties, you make it a contained member variable.
I wish I had a succinct explanation, but I think it really gets into the
nitty gritty details of how these serializer works (and this just a hunch
of mine anyway, I'm not 100% sure). Would be great if you could confirm
either way.
thanks,
Imran
On Tue, Apr 7, 2015 at 9:29 AM, Shuai Zheng <szheng.c...@gmail.com> wrote:
> I have found the issue, but I think it is bug.
>
>
>
> If I change my class to:
>
>
>
> public class ModelSessionBuilder implements Serializable {
>
> /**
>
> *
>
> */
>
> …
>
> private *Properties[] propertiesList*;
>
> private static final long serialVersionUID =
> -8139500301736028670L;
>
> }
>
>
>
> The broadcast value has no issue. But in my original form, if I broadcast
> it as array of my custom subclass of Properties, after broadcast, the
> propertiesList array will be an array of empty PropertiesUtils objects
> there (empty, not NULL), I am not sure why this happen (the code without
> any problem when run with default java serializer). So I think this is a
> bug, but I am not sure it is a bug of spark or a bug of Kryo.
>
>
>
> Regards,
>
>
>
> Shuai
>
>
>
>
> *From:* Shuai Zheng [mailto:szheng.c...@gmail.com]
> *Sent:* Monday, April 06, 2015 5:34 PM
> *To:* user@spark.apache.org
> *Subject:* Broadcast value return empty after turn to
> org.apache.spark.serializer.KryoSerializer
>
>
>
> Hi All,
>
>
>
> I have tested my code without problem on EMR yarn (spark 1.3.0) with
> default serializer (java).
>
> But when I switch to org.apache.spark.serializer.KryoSerializer, the
> broadcast value doesn’t give me right result (actually return me empty
> custom class on inner object).
>
>
>
> Basically I broadcast a builder object, which carry an array of
> propertiesUtils object. The code should not have any logical issue because
> it works on default java serializer. But when I turn to the
> org.apache.spark.serializer.KryoSerializer, it looks like the Array doesn’t
> initialize, propertiesList will give a right size, but then all element in
> the array is just a normal empty PropertiesUtils.
>
>
>
> Do I miss anything when I use this KryoSerializer? I just put the two
> lines, do I need to implement some special code to enable KryoSerializer,
> but I search all places but can’t find any places mention it.
>
>
>
> sparkConf.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer");
>
> sparkConf.registerKryoClasses(*new* Class[]{ModelSessionBuilder.*class*,
> Constants.*class*, PropertiesUtils.*class*, ModelSession.*class*});
>
>
>
> public class ModelSessionBuilder implements Serializable {
>
> /**
>
> *
>
> */
>
> …
>
> private PropertiesUtils[] propertiesList;
>
> private static final long serialVersionUID =
> -8139500301736028670L;
>
> }
>
>
>
> *public* *class* PropertiesUtils *extends* Properties {
>
> /**
>
> *
>
> */
>
> *private* *static* *final* *long* *serialVersionUID* =
> -3684043338580885551L;
>
>
>
> *public* PropertiesUtils(Properties prop) {
>
> *super*(prop);
>
> }
>
>
>
> *public* PropertiesUtils() {
>
> // *TODO* Auto-generated constructor stub
>
> }
>
> }
>
>
>
>
>
> Regards,
>
>
>
> Shuai
>
