Questions related to writing data to S3

2017-04-23 Thread Richard Hanson
I have a streaming job which writes data to S3. I know there are saveAs functions helping write data to S3. But it bundles all elements then writes out to S3. So my first question - Is there any way to let saveAs functions write data in batch or single elements instead of whole bundle?

Cannot convert from JavaRDD to Dataframe

2017-04-23 Thread Chen, Mingrui
Hello everyone! I am a new Spark learner and trying to do a task seems very simple. I want to read a text file, save the content to JavaRDD and convert it to Dataframe, so I can use it for Word2Vec Model in the future. The code looks pretty simple but I cannot make it work: SparkSession