Ok, great, I'm gonna do do it that way, thanks :). However I still don't understand why this object should be serialized and shipped?
aaa.s and sc are both the same object org.apache.spark.SparkContext@1f222881 However this : aaa.s.parallelize(1 to 10).filter(_ == myNumber).count Needs to be serialized, and this: sc.parallelize(1 to 10).filter(_ == myNumber).count does not. 2014-11-24 23:13 GMT+01:00 Marcelo Vanzin [via Apache Spark User List] < ml-node+s1001560n19692...@n3.nabble.com>: > On Mon, Nov 24, 2014 at 1:56 PM, aecc <[hidden email] > <http://user/SendEmail.jtp?type=node&node=19692&i=0>> wrote: > > I checked sqlContext, they use it in the same way I would like to use my > > class, they make the class Serializable with transient. Does this > affects > > somehow the whole pipeline of data moving? I mean, will I get > performance > > issues when doing this because now the class will be Serialized for some > > reason that I still don't understand? > > If you want to do the same thing, your "AAA" needs to be serializable > and you need to mark all non-serializable fields as "@transient". The > only performance penalty you'll be paying is the serialization / > deserialization of the "AAA" instance, which most probably will be > really small compared to the actual work the task will be doing. > > Unless your class is holding a whole lot of data, in which case you > should start thinking about using a broadcast instead. > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] > <http://user/SendEmail.jtp?type=node&node=19692&i=1> > For additional commands, e-mail: [hidden email] > <http://user/SendEmail.jtp?type=node&node=19692&i=2> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/Using-Spark-Context-as-an-attribute-of-a-class-cannot-be-used-tp19668p19692.html > To unsubscribe from Using Spark Context as an attribute of a class cannot > be used, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=19668&code=YWxlc3NhbmRyb2FlY2NAZ21haWwuY29tfDE5NjY4fDE2MzQ0ODgyMDU=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- Alessandro Chacón Aecc_ORG -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-Spark-Context-as-an-attribute-of-a-class-cannot-be-used-tp19668p19694.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
