Re: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-25 Thread Felix Cheung
eterList, scoreModel) Could you provide more information on your actual code? _ From: Cinquegrana, Piero <piero.cinquegr...@neustar.biz<mailto:piero.cinquegr...@neustar.biz>> Sent: Wednesday, August 24, 2016 10:37 AM Subject: RE: spark.lapply in SparkR: Error in wri

RE: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-25 Thread Cinquegrana, Piero
z>; user@spark.apache.org Subject: Re: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big") Hmm <<-- wouldn't work in cluster mode. Are you running spark in local mode? In any case, I tried running your earlier code and it worked for me on a 25

Re: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-25 Thread Felix Cheung
Cinquegrana, Piero <piero.cinquegr...@neustar.biz<mailto:piero.cinquegr...@neustar.biz>> Sent: Wednesday, August 24, 2016 10:37 AM Subject: RE: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big") To: Cinquegrana, Piero <piero.cinquegr...@neustar.biz<mailto:

RE: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-24 Thread Cinquegrana, Piero
day, August 23, 2016 2:39 PM To: Felix Cheung <felixcheun...@hotmail.com>; user@spark.apache.org Subject: RE: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big") The output from score() is very small, just a float. The input, however, could be as big as several hundre

RE: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-23 Thread Cinquegrana, Piero
, Piero <piero.cinquegr...@neustar.biz>; user@spark.apache.org Subject: Re: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big") How big is the output from score()? Also could you elaborate on what you want to broadcast? On Mon, Aug 22, 2016 at 11:58 AM -0700, &qu

Re: spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-22 Thread Felix Cheung
How big is the output from score()? Also could you elaborate on what you want to broadcast? On Mon, Aug 22, 2016 at 11:58 AM -0700, "Cinquegrana, Piero" > wrote: Hello, I am using the new R API in SparkR spark.lapply

spark.lapply in SparkR: Error in writeBin(batch, con, endian = "big")

2016-08-22 Thread Cinquegrana, Piero
Hello, I am using the new R API in SparkR spark.lapply (spark 2.0). I am defining a complex function to be run across executors and I have to send the entire dataset, but there is not (that I could find) a way to broadcast the variable in SparkR. I am thus reading the dataset in each executor