Hi Josh,
I did make that change, however I get this error now:
568.492: [GC [PSYoungGen: 1412948K-207017K(1465088K)]
4494287K-3471149K(4960384K), 0.1280200 secs] [Times: user=0.23 sys=0.63,
real=0.13 secs]
568.642: [Full GCTraceback (most recent call last):
File stdin, line 1, in module
File /home/hadoop/spark-1.0.1-bin-hadoop1/python/pyspark/rdd.py, line
708, in count
return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
File /home/hadoop/spark-1.0.1-bin-hadoop1/python/pyspark/rdd.py, line
699, in sum
return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)
File /home/hadoop/spark-1.0.1-bin-hadoop1/python/pyspark/rdd.py, line
619, in reduce
vals = self.mapPartitions(func).collect()
File /home/hadoop/spark-1.0.1-bin-hadoop1/python/pyspark/rdd.py, line
583, in collect
bytesInJava = self._jrdd.collect().iterator()
File /home/hadoop/spark-1.0.1-bin-hadoop1/python/pyspark/rdd.py, line
94, in __exit__
self._context._jsc.setCallSite(None)
File
/home/hadoop/spark-1.0.1-bin-hadoop1/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py,
line 535, in __call__
File
/home/hadoop/spark-1.0.1-bin-hadoop1/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py,
line 361, in send_command
File
/home/hadoop/spark-1.0.1-bin-hadoop1/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py,
line 317, in _get_connection
File
/home/hadoop/spark-1.0.1-bin-hadoop1/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py,
line 324, in _create_connection
File
/home/hadoop/spark-1.0.1-bin-hadoop1/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py,
line 431, in start
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect
to the Java server
*Vedant Dhandhania*
*Retention** Science*
call: 805.574.0873
visit: Site http://www.retentionscience.com/ | like: Facebook
http://www.facebook.com/RetentionScience | follow: Twitter
http://twitter.com/RetentionSci
On Fri, Jul 18, 2014 at 3:10 PM, Josh Rosen rosenvi...@gmail.com wrote:
You have to use `myBroadcastVariable.value` to access the broadcasted
value; see
https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables
On Fri, Jul 18, 2014 at 2:56 PM, Vedant Dhandhania
ved...@retentionscience.com wrote:
Hi All,
I am trying to broadcast a set in a PySpark script.
I create the set like this:
Uid_male_set = set(maleUsers.map(lambda x:x[1]).collect())
Then execute this line:
uid_iid_iscore_tuple_GenderFlag = uid_iid_iscore.map(lambda
x:(x[0],zip(x[1],x[2]),x[0] in Uid_male_set))
An error occurred while calling o104.collectPartitions.
: org.apache.spark.SparkException: Job aborted due to stage failure:
Serialized task 1131:0 was 23503247 bytes which exceeds
spark.akka.frameSize (10485760 bytes). Consider using broadcast variables
for large values.
So I tried broadcasting it:
Uid_male_setbc = sc.broadcast(Uid_male_set)
Uid_male_setbc
pyspark.broadcast.Broadcast object at 0x1ba2ed0
Then I execute it line:
uid_iid_iscore_tuple_GenderFlag = uid_iid_iscore.map(lambda
x:(x[0],zip(x[1],x[2]),x[0] in Uid_male_setbc))
ile stdin, line 1, in lambda
TypeError: argument of type 'Broadcast' is not iterable
[duplicate 1]
So I am stuck either ways, the script runs locally well on a smaller
dataset, but throws me this error. Could any one point out how to correct
this or where I am going wrong?
Thanks
*Vedant Dhandhania*
*Retention** Science*
call: 805.574.0873
visit: Site http://www.retentionscience.com/ | like: Facebook
http://www.facebook.com/RetentionScience | follow: Twitter
http://twitter.com/RetentionSci
