can't get around this error when performing union of two datasets (ds1.union(ds2)) having complex data type (struct, list),
*18/06/02 15:12:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. array<struct<id:int,booking_id:int,shifting_status:int,shifting_reason:int,shifting_metadata:string>> <> array<struct<id:int,booking_id:int,shifting_status:int,shifting_reason:int,shifting_metadata:string>> at the 21th column of the second table;;* As far as I can tell, they are the same. What am I doing wrong? Any help / workaround appreciated! spark version: 2.2.1 Thanks, Pranav
