voonhous commented on code in PR #18961:
URL: https://github.com/apache/hudi/pull/18961#discussion_r3820923846
##########
hudi-common/src/main/java/org/apache/hudi/common/avro/VariantSchemaUtils.java:
##########
@@ -123,6 +143,228 @@ private static HoodieSchema
stripVariantShreddingAt(HoodieSchema schema) {
return wasNullable ? HoodieSchema.createNullable(replacement) :
replacement;
}
+ /**
+ * Strips {@code typed_value} from top-level fields that have the variant
SHAPE but lost the
+ * variant logical type, i.e. plain records of {@code {metadata: bytes,
value: [nullable]
+ * bytes, typed_value}} (see {@link #isShreddedVariantShape}).
Parquet-footer-derived schemas
+ * come back this way (the converter does not attach the variant logical
type), so
+ * {@link #stripVariantShredding} alone cannot see them. Used by the
table-schema footer
+ * fallback only; returns {@code schema} as-is when nothing matches.
+ *
+ * <p>Unlike {@link #isShreddedVariantTarget}, the match here has NO
requested-side anchor:
+ * the footer fallback runs precisely when no table schema is available to
anchor on, so a
+ * plain user struct that happens to have exactly this shape is stripped too
(a documented,
+ * accepted false positive: {@code metadata} plus {@code typed_value} is the
variant spec's
+ * vocabulary). Top-level fields only, matching the scope of
+ * {@link #getInferableVariantColumns}: inference never shreds a nested
variant, and the
+ * forced-shredding hooks of both write supports are top-level too.
+ *
+ * <p>The shape check also admits the spec's two-field {@code {metadata,
typed_value}} form (a
+ * writer may omit {@code value} when every row is typed). Stripping that
would leave a
+ * one-field record, so {@code value} is restored as nullable bytes: the
result is always the
+ * unshredded {@code {metadata, value}} shape.
+ */
+ public static HoodieSchema stripVariantShreddingByShape(HoodieSchema schema)
{
+ if (schema.getType() != HoodieSchemaType.RECORD) {
+ return schema;
+ }
+
+ List<HoodieSchemaField> newFields = new ArrayList<>();
+ boolean changed = false;
+
+ for (HoodieSchemaField field : schema.getFields()) {
+ HoodieSchema fieldSchema = field.schema();
+ boolean wasNullable = fieldSchema.isNullable();
+ HoodieSchema unwrapped = wasNullable ? fieldSchema.getNonNullType() :
fieldSchema;
+
+ if (isShreddedVariantShape(unwrapped)) {
+ List<HoodieSchemaField> strippedFields = new ArrayList<>();
+ for (HoodieSchemaField member : unwrapped.getFields()) {
+ if
(!HoodieSchema.Variant.VARIANT_TYPED_VALUE_FIELD.equals(member.name())) {
+ strippedFields.add(HoodieSchemaUtils.createNewSchemaField(member));
+ }
+ }
+ if
(!unwrapped.getField(HoodieSchema.Variant.VARIANT_VALUE_FIELD).isPresent()) {
+ strippedFields.add(HoodieSchemaField.of(
+ HoodieSchema.Variant.VARIANT_VALUE_FIELD,
HoodieSchema.createNullable(HoodieSchemaType.BYTES)));
Review Comment:
Fixed in 5c2b9eed298e with the 4-arg overload passing
`HoodieSchema.NULL_VALUE`;
`testStripVariantShreddingByShapeRestoresOmittedValue` now pins the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
