Use a tool like flume and/or oozie to reliable download files from http and do
error handling (e.g. Requests time out). Afterwards process the data with spark.
> On 25 Sep 2016, at 10:27, Dan Bikle wrote:
>
> hello spark-world,
>
> How to use Spark-Scala to download a CSV fi
Hi
not sure if spark-csv supports the http:// format you use to load data
from the WEB. I just tried this and got exception
scala> val df = sqlContext.read.
| format("com.databricks.spark.csv").
| option("inferSchema", "true").
| load("http://ichart.finance.yahoo.com/table.csv?s=F
hello spark-world,
How to use Spark-Scala to download a CSV file from the web and load the
file into a spark-csv DataFrame?
Currently I depend on curl in a shell command to get my CSV file.
Here is the syntax I want to enhance:
*/* fb_csv.scalaThis script should load FB
