Spark 2.0 Encoder().schema() is sorting StructFields

2016-10-12 Thread Paul Stewart
Hi all, I am using Spark 2.0 to read a CSV file into a Dataset in Java. This works fine if i define the StructType with the StructField array ordered by hand. What I would like to do is use a bean class for both the schema and Dataset row type. For example, Dataset beanDS = spark.read().sch

Spark 2.0 Encoder().schema() is sorting StructFields

2016-10-07 Thread Paul Stewart
When using the Encoder(Bean.class).schema() method to generate the StructType array of StructFields the class attributes are returned as a sorted list and not in the defined order within the Bean.class. This makes the schema unusable for reading from a CSV file where the ordering of the attribute