Thanks Vijay. After changing the programming model (create a context class for the workers), it finally worked for me. Cheers.
On Fri, Feb 23, 2018 at 5:42 PM, vijay.bvp <bvpsa...@gmail.com> wrote: > when HTTP connection is opened you are opening a connection between > specific > machine (with IP and NIC card) to another specific machine, so this can't > be > serialized and used on other machine right!! > > This isn't spark limitation. > > I made a simple diagram if it helps. The Objects created at driver and > passed to worker need to be serialized. The objects created at workers need > not. > > In the diagram you have to create HTTPConnection on the executors > independently of the driver. > The HTTPConnection created at Executor-1 can be used for partitions P1-P3 > of > RDD available on that executor. > > Spark is tolerant and does allow passing objects from driver to workers, > but > in case if it reports "Task not serializable" it does indicate some object > is having issue. mark the class as Serializable if you think if the object > of it can be serialized. As I said in the beginning not everything could > serializable particularly http connections, JDBC connections etc.. > > <http://apache-spark-user-list.1001560.n3.nabble.com/ > file/t8878/Picture1.png> > > > > > > > > > > > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
