I don't think Broadcast itself can be serialized. you can get the value
>> out on the driver side and refer to it in foreach, then the value would be
>> serialized with the lambda expr and sent to workers.
>>
>> On Fri, Jun 16, 2017 at 2:29 AM, Anton Kravchenko <
>>
think Broadcast itself can be serialized. you can get the value
> out on the driver side and refer to it in foreach, then the value would be
> serialized with the lambda expr and sent to workers.
>
> On Fri, Jun 16, 2017 at 2:29 AM, Anton Kravchenko <
> kravchenko
> How one would access a broadcasted variable from within
> ForeachPartitionFunction Spark(2.0.1) Java API ?
>
> Integer _bcv = 123;
> Broadcast bcv = spark.sparkContext().broadcast(_bcv);
> Dataset df_sql = spark.sql("select * from atable");
>
> df_sql.forea
How one would access a broadcasted variable from within
ForeachPartitionFunction Spark(2.0.1) Java API ?
Integer _bcv = 123;
Broadcast bcv = spark.sparkContext().broadcast(_bcv);
Dataset df_sql = spark.sql("select * from atable");
df_sql.foreachPartition(new ForeachPartiti
ore confused
> I thought it was supposed to save memory by distributing it to every
> worker and the executors would share that copy
>
>
> --
> Date: Fri, 19 Feb 2016 16:48:59 -0800
> Subject: Re: Access to broadcasted variable
> From: s
Is the broadcasted variable distributed to every executor or every worker? Now
i'm more confused
I thought it was supposed to save memory by distributing it to every worker and
the executors would share that copy
Date: Fri, 19 Feb 2016 16:48:59 -0800
Subject: Re: Access to broadcasted variable
s? thanks
>
> --
> From: jeffsar...@hotmail.com
> To: user@spark.apache.org
> Subject: Access to broadcasted variable
> Date: Thu, 18 Feb 2016 14:44:07 -0500
>
>
>
> I'd like to know if the broadcasted object gets serialized when accessed
> by the executo
could someone please comment on this? thanks
From: jeffsar...@hotmail.com
To: user@spark.apache.org
Subject: Access to broadcasted variable
Date: Thu, 18 Feb 2016 14:44:07 -0500
I'd like to know if the broadcasted object gets serialized when accessed by the
executor during the execution
I'd like to know if the broadcasted object gets serialized when accessed by the
executor during the execution of a task?
I know that it gets serialized from the driver to the worker. This question is
inside worker when executor JVM's are accessing it
thanks
Jeff
