Re: Switching broadcast mechanism from torrrent

I agree, it was by mistake. I just updated so that the next person looking for torrent broadcast issues will have a hint :) Thank you. Daniel On Sun, Jun 19, 2016 at 5:26 PM, Ted Yu wrote: > I think good practice is not to hold on to SparkContext in mapFunction. > > On

Re: Switching broadcast mechanism from torrrent

I think good practice is not to hold on to SparkContext in mapFunction. On Sun, Jun 19, 2016 at 7:10 AM, Takeshi Yamamuro wrote: > How about using `transient` annotations? > > // maropu > > On Sun, Jun 19, 2016 at 10:51 PM, Daniel Haviv < >

Re: Switching broadcast mechanism from torrrent

How about using `transient` annotations? // maropu On Sun, Jun 19, 2016 at 10:51 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > Just updating on my findings for future reference. > The problem was that after refactoring my code I ended up with a scala > object which held

Re: Switching broadcast mechanism from torrrent

Hi, Just updating on my findings for future reference. The problem was that after refactoring my code I ended up with a scala object which held SparkContext as a member, eg: object A { sc: SparkContext = new SparkContext def mapFunction {} } and when I called rdd.map(A.mapFunction) it

Re: Switching broadcast mechanism from torrrent

Hi, Since `HttpBroadcastFactory` has already been removed in master, so you cannot use the broadcast mechanism in future releases. Anyway, I couldn't find a root cause only from the stacktraces... // maropu On Mon, Jun 6, 2016 at 2:14 AM, Daniel Haviv < daniel.ha...@veracity-group.com>

Re: Switching broadcast mechanism from torrrent

Hi, I've set spark.broadcast.factory to org.apache.spark.broadcast.HttpBroadcastFactory and it indeed resolve my issue. I'm creating a dataframe which creates a broadcast variable internally and then fails due to the torrent broadcast with the following stacktrace: Caused by:

Re: Switching broadcast mechanism from torrrent

I found spark.broadcast.blockSize but no parameter to switch broadcast method. Can you describe the issues with torrent broadcast in more detail ? Which version of Spark are you using ? Thanks On Wed, Jun 1, 2016 at 7:48 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > Our

Switching broadcast mechanism from torrrent

Hi, Our application is failing due to issues with the torrent broadcast, is there a way to switch to another broadcast method ? Thank you. Daniel