Ok thanks for the help! Best, Flavio
On Tue, Apr 15, 2014 at 12:43 AM, Eugen Cepoi <cepoi.eu...@gmail.com> wrote: > Nope, those operations are lazy, meaning it will create the RDDs but won't > trigger any "action". The computation is launched by operations such as > collect, count, save to HDFS etc. And even if they were not lazy, no > serialization would happen. Serialization occurs only when data will be > transfered (collect, shuffle, maybe perist to disk - but I am not sure for > this one). > > > 2014-04-15 0:34 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: > > Ok, that's fair enough. But why things work up to the collect?during map >> and filter objects are not serialized? >> On Apr 15, 2014 12:31 AM, "Eugen Cepoi" <cepoi.eu...@gmail.com> wrote: >> >>> Sure. As you have pointed, those classes don't implement Serializable >>> and Spark uses by default java serialization (when you do collect the data >>> from the workers will be serialized, "collected" by the driver and then >>> deserialized on the driver side). Kryo (as most other decent serialization >>> libs) doesn't require you to implement Serializable. >>> >>> For the missing attributes it's due to the fact that java serialization >>> does not ser/deser attributes from classes that don't impl. Serializable >>> (in your case the parent classes). >>> >>> >>> 2014-04-14 23:17 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: >>> >>>> Thanks Eugen for tgee reply. Could you explain me why I have the >>>> problem?Why my serialization doesn't work? >>>> On Apr 14, 2014 6:40 PM, "Eugen Cepoi" <cepoi.eu...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> as a easy workaround you can enable Kryo serialization >>>>> http://spark.apache.org/docs/latest/configuration.html >>>>> >>>>> Eugen >>>>> >>>>> >>>>> 2014-04-14 18:21 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: >>>>> >>>>>> Hi to all, >>>>>> >>>>>> in my application I read objects that are not serializable because I >>>>>> cannot modify the sources. >>>>>> So I tried to do a workaround creating a dummy class that extends the >>>>>> unmodifiable one but implements serializable. >>>>>> All attributes of the parent class are Lists of objects (some of them >>>>>> are still not serializable and some of them are, i.e. List<String>). >>>>>> >>>>>> Until I do map and filter on the RDD that objects are filled >>>>>> correclty (I checked that via Eclipse debug), but when I do collect all >>>>>> the >>>>>> attributes of my objects are empty. Could you help me please? >>>>>> I'm using spark-core-2.10 e version 0.9.0-incubating. >>>>>> >>>>>> Best, >>>>>> Flavio >>>>>> >>>>>> >>>>> >>>
