yes, issue is with array type only, I have confirmed that.
I exploded array to struct but still getting the same error,
*Exception in thread "main" org.apache.spark.sql.AnalysisException: Union
can only be performed on tables with the compatible column types.
struct
<>
struct
at the 21th column
Have you tryed to narrow down the problem so that we can be 100% sure that it
lies on the array types ? Just exclude them for sake of testing.
If we know 100% that it is on this array stuff try to explode that columns into
simple types.
Jorge Machado
> On 4 Jun 2018, at 11:09, Pranav
I am ordering the columns before doing union, so I think that should not be
an issue,
* String[] columns_original_order = baseDs.columns();
String[] columns = baseDs.columns();Arrays.sort(columns);
baseDs=baseDs.selectExpr(columns);
Try the same union with a dataframe without the arrays types. Could be
something strange there like ordering or so.
Jorge Machado
> On 4 Jun 2018, at 10:17, Pranav Agrawal wrote:
>
> schema is exactly the same, not sure why it is failing though.
>
> root
> |-- booking_id: integer
schema is exactly the same, not sure why it is failing though.
root
|-- booking_id: integer (nullable = true)
|-- booking_rooms_room_category_id: integer (nullable = true)
|-- booking_rooms_room_id: integer (nullable = true)
|-- booking_source: integer (nullable = true)
|-- booking_status:
Hi Pranav,
I don´t have an answer to your issue, but what I generally do in this cases
is to first try to simplify it to a point where it is easier to check
what´s going on, and then adding back ¨pieces¨ one by one until I spot the
error.
In your case I can suggest to:
1) project the dataset to
can't get around this error when performing union of two datasets
(ds1.union(ds2)) having complex data type (struct, list),
*18/06/02 15:12:00 INFO ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception:
org.apache.spark.sql.AnalysisException: Union can
can't get around this error when performing union of two datasets having
complex data type (struct, list),
*18/06/02 15:12:00 INFO ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception:
org.apache.spark.sql.AnalysisException: Union can only be performed
