Re: how to load compressed (gzip) csv file using spark-csv

Thanks. It works. On Thu, Jun 16, 2016 at 5:32 PM Hyukjin Kwon wrote: > It will 'auto-detect' the compression codec by the file extension and then > will decompress and read it correctly. > > Thanks! > > 2016-06-16 20:27 GMT+09:00 Vamsi Krishna : >

Re: how to load compressed (gzip) csv file using spark-csv

It will 'auto-detect' the compression codec by the file extension and then will decompress and read it correctly. Thanks! 2016-06-16 20:27 GMT+09:00 Vamsi Krishna : > Hi, > > I'm using Spark 1.4.1 (HDP 2.3.2). > As per the spark-csv documentation ( >

how to load compressed (gzip) csv file using spark-csv

Hi, I'm using Spark 1.4.1 (HDP 2.3.2). As per the spark-csv documentation (https://github.com/databricks/spark-csv), I see that we can write to a csv file in compressed form using the 'codec' option. But, didn't see the support for 'codec' option to read a csv file. Is there a way to read a