I need to use few columns out of a csv. But as there is no option to read few columns out of csv so 1. I am reading the whole CSV using SparkSession.csv() 2. selecting few of the columns using DataFrame.select() 3. applying schema using the .as() function of Dataset<Row>. I tried to extent org.apache.spark.sql.Encoder as the input for as function
But I am getting the following exception Exception in thread "main" java.lang.RuntimeException: Only expression encoders are supported today So my questions are - 1. Is it possible to read few columns instead of whole CSV? I cannot change the CSV as that is upstream data 2. How do I apply schema to few columns if I cannot write my encoder?
