Re: Spark 2.0 Shell -csv package weirdness

2016-03-20 Thread Vincent Ohprecio
ws > > Started at > [20/03/2016 10:02:02.02] > > Elapsed time: 63984582000ns > howLong: Unit = () > Finished at > [20/03/2016 10:04:59.59] > > So the whole job finished just under 3 minutes. The elapsed time for > saving output.csv took 63 seconds. That CSV file has 7,

Fwd: Spark 2.0 Shell -csv package weirdness

2016-03-19 Thread Vincent Ohprecio
For some reason writing data from Spark shell to csv using the `csv package` takes almost an hour to dump to disk. Am I going crazy or did I do this wrong? I tried writing to parquet first and its fast as normal. On my Macbook Pro 16g - 2.2 GHz Intel Core i7 -1TB the machine CPU's goes crazy and