Re: how to skip headers when reading multiple files

Are you read csv file ? If so you can use spark-csv which support skip header http://spark-packages.org/package/databricks/spark-csv On Thu, Dec 3, 2015 at 10:52 AM, Divya Gehlot wrote: > Hi, > I am new bee to Spark and Scala . > As one of my requirement to read and

Re: how to skip headers when reading multiple files

You could use "filter" to eliminate headers from your text file RDD while going over each line. -Sahil On Thu, Dec 3, 2015 at 9:37 AM, Jeff Zhang wrote: > Are you read csv file ? If so you can use spark-csv which support skip > header > >

how to skip headers when reading multiple files

Hi, I am new bee to Spark and Scala . As one of my requirement to read and process multiple text files with headers using DataFrame API . How can I skip headers when processing data with DataFrame API Thanks in advance . Regards, Divya