Hi all,
I'm using Spark SQL using data from Openstack Swift.
I'm trying to load parquet files with partition discovery, but I can't do
it when the partitions don't match between two objects.
For example, container which contains:
/zone=0/object2
/zone=0/area=0/object1
Won't load, and will result with the error:
java.lang.AssertionError: assertion failed: Conflicting partition column
names detected:
ArrayBuffer(zone, area)
ArrayBuffer(zone)
I know that there is support for schema evolution for the schemas inside
the parquet files, but is there a schema evolution for partition discovery
as well?
Guy Hadash
IBM Research - Haifa
03-7689436
