It’s true that it can’t. You can try to use the CloudPickle library instead,
which is what we use within PySpark to serialize functions (see
python/pyspark/cloudpickle.py). However I’m also curious, why do you need an
RDD of functions?
Matei
On Jun 15, 2014, at 4:49 PM, madeleine
.1001560.n3.nabble.com/pyspark-serializer-can-t-handle-functions-tp7650p7682.html
To unsubscribe from pyspark serializer can't handle functions?, click
here
http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=7650code
.
If you reply to this email, your message will be added to the discussion
below:
http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-serializer-can-t-handle-functions-tp7650p7682.html
To unsubscribe from pyspark serializer can't handle functions?, click here.
NAML
--
Madeleine Udell
It seems that the default serializer used by pyspark can't serialize a list
of functions.
I've seen some posts about trying to fix this by using dill to serialize
rather than pickle.
Does anyone know what the status of that project is, or whether there's
another easy workaround?
I've pasted a
