<a...@santacruzintegration.com>
Date: Wednesday, July 27, 2016 at 4:25 PM
To: "user @spark" <user@spark.apache.org>
Subject: how to copy local files to hdfs quickly?
> I have a spark streaming app that saves JSON files to s3:// . It works fine
>
> Now I need to calculate
I have a spark streaming app that saves JSON files to s3:// . It works fine
Now I need to calculate some basic summary stats and am running into
horrible performance problems.
I want to run a test to see if reading from hdfs instead of s3 makes
difference. I am able to quickly copy my the data
