Hi,
Just ran your code on spark-shell. If you replace
val bcA = sc.broadcast(a)
with
val bcA = sc.broadcast(new B().getA)
it seems to work. Not sure why.
On Tue, Dec 23, 2014 at 9:12 AM, Henry Hung <ythu...@winbond.com> wrote:
> Hi All,
>
>
>
> I have a problem with broadcasting a serialize class object that returned
> by another not-serialize class, here is the sample code:
>
>
>
> class A extends java.io.Serializable {
>
> def halo(): String = "halo"
>
> }
>
>
>
> class B {
>
> def getA() = new A
>
> }
>
>
>
> val list = List(1)
>
>
>
> val b = new B
>
> val a = b.getA
>
>
>
> val p = sc.parallelize(list)
>
>
>
> // this will fail
>
> val bcA = sc.broadcast(a)
>
> p.map(x => {
>
> bcA.value.halo()
>
> })
>
>
>
> // this will success
>
> val bcA = sc.broadcast(new A)
>
> p.map(x => {
>
> bcA.value.halo()
>
> })
>
>
>
>
>
> A is a serializable class, where B is not-serialize.
>
> If I create a new object A through B method getA(), the map process will
> failed with exception “org.apache.spark.SparkException: Task not
> serializable, Caused by: java.io.NotSerializableException: $iwC$$iwC$B”
>
>
>
> I don’t know why spark will check if the B class serializable or not, is
> there a way to code this?
>
>
>
> Best regards,
>
> Henry
>
> ------------------------------
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original sender
> of this email. If you are not the addressee indicated in this email or are
> not responsible for delivery of the email to such a person, please kindly
> reply to the sender indicating this fact and delete all copies of it from
> your computer and network server immediately. Your cooperation is highly
> appreciated. It is advised that any unauthorized use of confidential
> information of Winbond is strictly prohibited; and any information in this
> email irrelevant to the official business of Winbond shall be deemed as
> neither given nor endorsed by Winbond.
>
--
Regards,
Madhukara Phatak
http://www.madhukaraphatak.com
