Spark with GCS Connector - Rate limit error

Hi, I'm using Spark on a Google Compute Engine cluster with the Google Cloud Storage connector (instead of HDFS, as recommended here https://cloud.google.com/hadoop/google-cloud-storage-connector#benefits ), and get a lot of rate limit errors, as added below. The errors relate to temp files

Re: Writing binary files in Spark

://issues.apache.org/jira/browse/SPARK-1161 Thanks Best Regards On Sat, Jul 25, 2015 at 12:06 AM, Oren Shpigel o...@yowza3d.com wrote: Sorry, I didn't mention I'm using the Python API, which doesn't have the saveAsObjectFiles method. Is there any alternative from Python? And also, I want to write

Re: Writing binary files in Spark

alternatives for that? Thanks, Oren On Thu, Jul 23, 2015 at 8:04 PM Akhil Das ak...@sigmoidanalytics.com wrote: You can look into .saveAsObjectFiles Thanks Best Regards On Thu, Jul 23, 2015 at 8:44 PM, Oren Shpigel o...@yowza3d.com wrote: Hi, I use Spark to read binary files using

Writing binary files in Spark

Hi, I use Spark to read binary files using SparkContext.binaryFiles(), and then do some calculations, processing, and manipulations to get new objects (also binary). The next thing I want to do is write the results back to binary files on disk. Is there any equivalence like saveAsTextFile just