difin commented on code in PR #6927:
URL: https://github.com/apache/hive/pull/6927#discussion_r4622726937
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/BaseHiveIcebergMetaHook.java:
##########
@@ -453,6 +456,45 @@ private Schema getSchemaWithIdentifierFields(Schema
schema, Set<String> identifi
return new Schema(cols, identifierFieldIds);
}
+ /**
+ * UNKNOWN values are omitted from data files, so Iceberg can only write
them as top-level columns or as struct
+ * fields that have at least one sibling of another type. Lists, maps and
all-UNKNOWN structs would have no
+ * physical representation left.
+ */
+ static void validateUnknownType(String columnPath, Type type) {
+ switch (type.typeId()) {
+ case STRUCT -> {
+ List<Types.NestedField> fields = type.asStructType().fields();
+ Preconditions.checkArgument(fields.stream().anyMatch(field ->
!isUnknown(field.type())),
+ "Column '%s' of type %s is not supported: a struct must have at
least one field that is not of type " +
Review Comment:
Creating it works, but the table isn't usable on every format. UNKNOWN
columns aren't written to data files, so a table whose columns are all UNKNOWN
has an empty file schema. Parquet refuses to write it ("Cannot write a schema
with an empty group"), and Hive's vectorized ORC reader can't open it. Avro and
non-vectorized ORC work. This is the same reason structs whose fields are all
UNKNOWN are already rejected.
I'll reject it at DDL time instead: CREATE TABLE, REPLACE COLUMNS and DROP
COLUMN will fail with "a table must have at least one column that is not of
type unknown". I'll add a negative q-test for it. A lone UNKNOWN column next to
other columns stays supported, and `positive/unknown_type.q` covers that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]