Ah, good to know! By the way in master we now have saveAsPickleFile (https://github.com/apache/spark/pull/755), and Nick Pentreath has been working on Hadoop InputFormats: https://github.com/apache/spark/pull/455. Would be good to have your input on both of those if you have a chance to try them.
Matei On Jun 4, 2014, at 3:28 PM, Jeremy Freeman <freeman.jer...@gmail.com> wrote: > Hey Matei, > > Wanted to let you know this issue appears to be fixed in 1.0.0. Great work! > > -- Jeremy > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/error-loading-large-files-in-PySpark-0-9-0-tp3049p6985.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
