Hey Bing,
There’s a couple different approaches you could take. The quickest and easiest
would be to use the existing APIs
val bytes = spark.range(1000
bytes.foreachPartition(bytes =>{
//W ARNING anything used in here will need to be serializable.
// There's some magic to serializing the hadoop conf. see the hadoop wrapper
class in the source
val writer = FileSystem.get(null).create(new Path("s3://..."))
bytes.foreach(b => writer.write(b))
writer.close()
})
The more complicated but pretty approach would be to either implement a custom
datasource.
From: "Duan,Bing" <duanb...@baidu.com>
Date: Thursday, January 16, 2020 at 12:35 AM
To: "dev@spark.apache.org" <dev@spark.apache.org>
Subject: How to implement a "saveAsBinaryFile" function?
Hi all:
I read binary data(protobuf format) from filesystem by binaryFiles function to
a RDD[Array[Byte]] it works fine. But when I save the it to filesystem by
saveAsTextFile, the quotation mark was be escaped like this:
"\"201900002_1\"",1,24,0,2,"\"S66.000x001\””, which should be
"201900002_1",1,24,0,2,”S66.000x001”.
Anyone could give me some tip to implement a function like saveAsBinaryFile to
persist the RDD[Array[Byte]]?
Bests!
Bing
