Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5539652823 ## CI report: * c803cdca6c03e279852afe3f8f5fe11e16363d07 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16827) * 27cbba3e3ab57a6704aa294158a7a9df1b751f91 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous merged PR #19810: URL: https://github.com/apache/hudi/pull/19810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5538942257 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) * c803cdca6c03e279852afe3f8f5fe11e16363d07 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16827) * 27cbba3e3ab57a6704aa294158a7a9df1b751f91 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5538745123 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) * c803cdca6c03e279852afe3f8f5fe11e16363d07 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16827) * 27cbba3e3ab57a6704aa294158a7a9df1b751f91 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5538207531 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) * c803cdca6c03e279852afe3f8f5fe11e16363d07 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16827) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5538142776 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) * c803cdca6c03e279852afe3f8f5fe11e16363d07 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3932397785
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaCompatibility.java:
##
@@ -701,6 +703,155 @@ public void testIsSchemaCompatibleWithTypePromotion() {
assertFalse(HoodieSchemaCompatibility.isSchemaCompatible(longS, intS,
true, true));
}
+ /**
+ * Sibling of {@link #testIsSchemaCompatibleWithTypePromotion()} covering
the rest of the reader/writer type
+ * table: the primitive widening cases shared with {@link
HoodieSchemaTypePromotion}, plus the two
+ * logical-type-over-primitive rules (TIMESTAMP over LONG, UUID over STRING)
that are compatibility-only.
+ *
+ * All pairs are asserted through the 4-arg {@code
isSchemaCompatible(prev = writer, new = reader, true, true)}.
+ */
+ @Test
+ public void testIsSchemaCompatibleWithLogicalTypesAndWidening() {
+// Logical type over its backing primitive: accepted for reader/writer
compatibility.
+assertCompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertCompatible(HoodieSchema.createUUID(),
HoodieSchema.create(HoodieSchemaType.STRING));
+
+// ... but only in that direction, and only over the matching primitive.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.createTimestampMillis());
+assertIncompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.INT));
+// DATE has no such rule at all, even though it is backed by INT.
+assertIncompatible(HoodieSchema.createDate(),
HoodieSchema.create(HoodieSchemaType.INT));
+
+// Primitive widening, delegated to HoodieSchemaTypePromotion.
+assertCompatible(HoodieSchema.create(HoodieSchemaType.DOUBLE),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.FLOAT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+// The narrowing direction is rejected for every numeric pair. The
hudi-spark guard for the reversed
+// argument bug class (TestTableSchemaEvolution, HUDI-1493) never runs, so
the pairs are pinned here.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.BYTES));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.BYTES),
HoodieSchema.create(HoodieSchemaType.STRING));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.INT));
+ }
+
+ @Test
+ public void testAreSchemasCompatibleReaderIsFirstArgument() {
+HoodieSchema longRecord = HoodieSchemaTestUtils.createRecord("R",
HoodieSchemaField.of("f", HoodieSchema.create(HoodieSchemaType.LONG), null,
null));
+HoodieSchema intRecord = HoodieSchemaTestUtils.createRecord("R",
HoodieSchemaField.of("f", HoodieSchema.create(HoodieSchemaType.INT), null,
null));
+
+// A long reader can read int data ...
+assertTrue(HoodieSchemaCompatibility.areSchemasCompatible(longRecord,
intRecord));
+// ... but not the other way round, which pins the reader as the FIRST
argument.
+assertFalse(HoodieSchemaCompatibility.areSchemasCompatible(intRecord,
longRecord));
+ }
+
+ @Test
+ public void testLookupWriterFieldDirectMatch() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"a\",\"type\":\"int\"}]}");
+
+HoodieSchemaField writerField =
HoodieSchemaCompatibility.lookupWriterField(writerSchema, readerField);
+assertEquals("a", writerField.name());
+ }
+
+ /**
+ * The alias path is also driven end to end through the sole production
caller,
+ * {@code HoodieTable#validateSchema} (see {@code
TestHoodieTableSchemaEvolution#testFieldWithAlias}); this
Review Comment:
Both, in c803cdca6c03. The javadoc named the wrong method: the caller is
`validateSecondaryIndexSchemaEvolution`. And `testFieldWithAlias` now also
retypes `new_name` to `int` behind the same alias and expects
`SchemaCompatibilityException`, so an unresolved alias -- null writer field,
skipped by the `writerField != null` guard -- fails it. The javadoc points at
that half.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3932395704
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##
@@ -75,6 +75,17 @@ public class TestHoodieSchema {
+ " ]"
+ "}";
+ // The Blob schema is persisted in the table schema of every BLOB table, so
its serialized shape is pinned literally.
+ private static final String BLOB_SCHEMA_JSON =
"{\"type\":\"record\",\"name\":\"blob\",\"fields\":["
Review Comment:
Renamed to `EXPECTED_BLOB_SCHEMA_JSON` in c803cdca6c03.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810: URL: https://github.com/apache/hudi/pull/19810#discussion_r3932393742 ## hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaCompatibilityChecker.java: ## @@ -357,24 +357,24 @@ private SchemaCompatibilityResult calculateCompatibility(final HoodieSchema read case DATE: case DECIMAL: return result.mergedWith(typeMismatch(reader, writer, locations)); + // TIMESTAMP over LONG and UUID over STRING are reader/writer compatibility rules only. They are deliberately + // absent from HoodieSchemaTypePromotion: the projection checker must not treat a bare long as a compatible Review Comment: Right, the sentence had it backwards. Fixed in c803cdca6c03: both the checker comment and the `HoodieSchemaTypePromotion` javadoc now say the entry kept out is the one that would make a timestamp a compatible projection of a bare long, and they spell out `isCompatibleProjectionOf(source, target)` -> `canPromote(target, source)` so the direction is checkable from the text. The consequence is unchanged: `reconcileSchemasLegacy` would keep the table's long as the writer schema and the logical type would go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
wombatu-kun commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3930364615
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaCompatibilityChecker.java:
##
@@ -357,24 +357,24 @@ private SchemaCompatibilityResult
calculateCompatibility(final HoodieSchema read
case DATE:
case DECIMAL:
return result.mergedWith(typeMismatch(reader, writer, locations));
+ // TIMESTAMP over LONG and UUID over STRING are reader/writer
compatibility rules only. They are deliberately
+ // absent from HoodieSchemaTypePromotion: the projection checker
must not treat a bare long as a compatible
Review Comment:
`isCompatibleProjectionOf(source, target)` tests `canPromote(target,
source)`, so the entry kept out of the table is the one that would make a
timestamp a compatible projection of a bare long - this has it the other way
round. Could the direction be flipped here and in the matching sentence on
`HoodieSchemaTypePromotion`?
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaCompatibility.java:
##
@@ -701,6 +703,155 @@ public void testIsSchemaCompatibleWithTypePromotion() {
assertFalse(HoodieSchemaCompatibility.isSchemaCompatible(longS, intS,
true, true));
}
+ /**
+ * Sibling of {@link #testIsSchemaCompatibleWithTypePromotion()} covering
the rest of the reader/writer type
+ * table: the primitive widening cases shared with {@link
HoodieSchemaTypePromotion}, plus the two
+ * logical-type-over-primitive rules (TIMESTAMP over LONG, UUID over STRING)
that are compatibility-only.
+ *
+ * All pairs are asserted through the 4-arg {@code
isSchemaCompatible(prev = writer, new = reader, true, true)}.
+ */
+ @Test
+ public void testIsSchemaCompatibleWithLogicalTypesAndWidening() {
+// Logical type over its backing primitive: accepted for reader/writer
compatibility.
+assertCompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertCompatible(HoodieSchema.createUUID(),
HoodieSchema.create(HoodieSchemaType.STRING));
+
+// ... but only in that direction, and only over the matching primitive.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.createTimestampMillis());
+assertIncompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.INT));
+// DATE has no such rule at all, even though it is backed by INT.
+assertIncompatible(HoodieSchema.createDate(),
HoodieSchema.create(HoodieSchemaType.INT));
+
+// Primitive widening, delegated to HoodieSchemaTypePromotion.
+assertCompatible(HoodieSchema.create(HoodieSchemaType.DOUBLE),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.FLOAT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+// The narrowing direction is rejected for every numeric pair. The
hudi-spark guard for the reversed
+// argument bug class (TestTableSchemaEvolution, HUDI-1493) never runs, so
the pairs are pinned here.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.INT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.BYTES));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.BYTES),
HoodieSchema.create(HoodieSchemaType.STRING));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.INT));
+ }
+
+ @Test
+ public void testAreSchemasCompatibleReaderIsFirstArgument() {
+HoodieSchema longRecord = HoodieSchemaTestUtils.createRecord("R",
HoodieSchemaField.of("f", HoodieSchema.create(HoodieSchemaType.LONG), null,
null));
+HoodieSchema intRecord = HoodieSchemaTestUtils.createRecord("R",
HoodieSchemaField.of("f", HoodieSchema.create(HoodieSchemaType.INT), null,
null));
+
+// A long reader can read int data ...
+assertTrue(HoodieSchemaCompatibility.areSchemasCompatible(longRecord,
intRecord));
+// ... but not the other way round, which pins the reader as the FIRST
argument.
+assertFalse(HoodieSchemaCompatibility.areSchemasCompatible(intRecord,
longRecord));
+ }
+
+ @Test
+ public void testLookupWriterFieldDirectMatch() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-agent commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3927507710
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##
@@ -2953,23 +2954,34 @@ private static Schema createSchema(String name) {
return blobSchema;
}
-private static List createBlobFields() {
- Schema bytesField = Schema.create(Schema.Type.BYTES);
+private static Schema createReferenceSchema() {
Schema referenceField = Schema.createRecord(EXTERNAL_REFERENCE, null,
null, false);
List referenceFields = Arrays.asList(
new Schema.Field(EXTERNAL_REFERENCE_PATH,
Schema.create(Schema.Type.STRING), null, null),
- new Schema.Field(EXTERNAL_REFERENCE_OFFSET,
AvroSchemaUtils.createNullableSchema(Schema.create(Schema.Type.LONG)), null,
null),
- new Schema.Field(EXTERNAL_REFERENCE_LENGTH,
AvroSchemaUtils.createNullableSchema(Schema.create(Schema.Type.LONG)), null,
null),
+ new Schema.Field(EXTERNAL_REFERENCE_OFFSET,
nullable(Schema.create(Schema.Type.LONG)), null, null),
+ new Schema.Field(EXTERNAL_REFERENCE_LENGTH,
nullable(Schema.create(Schema.Type.LONG)), null, null),
new Schema.Field(EXTERNAL_REFERENCE_IS_MANAGED,
Schema.create(Schema.Type.BOOLEAN), null, null)
);
referenceField.setFields(referenceFields);
+ return referenceField;
+}
+private static List createBlobFields() {
+ Schema bytesField = Schema.create(Schema.Type.BYTES);
return Arrays.asList(
new Schema.Field(TYPE, Schema.createEnum("blob_storage_type", null,
null, Arrays.asList(INLINE, OUT_OF_LINE)), null, null),
- new Schema.Field(INLINE_DATA_FIELD,
AvroSchemaUtils.createNullableSchema(bytesField), null,
Schema.Field.NULL_DEFAULT_VALUE),
- new Schema.Field(EXTERNAL_REFERENCE,
AvroSchemaUtils.createNullableSchema(referenceField), null,
Schema.Field.NULL_DEFAULT_VALUE)
+ new Schema.Field(INLINE_DATA_FIELD, nullable(bytesField), null,
Schema.Field.NULL_DEFAULT_VALUE),
+ new Schema.Field(EXTERNAL_REFERENCE, nullable(REFERENCE_SCHEMA),
null, Schema.Field.NULL_DEFAULT_VALUE)
);
}
+
+/**
+ * Wraps the given schema into the canonical Avro nullable union {@code
[null, schema]}. None of the blob
+ * field types is NULL, so no further validation is needed here.
+ */
+private static Schema nullable(Schema schema) {
Review Comment:
🤖 nit: `nullable` reads as an adjective (a property check) rather than a
factory — could you rename it to `wrapNullable` or `createNullableUnion` so
call sites like `nullable(bytesField)` are self-explanatory?
⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5529696920 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5529775321 ## CI report: * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5526423894 ## CI report: * 4f9886f810fc14798e4d51b8cadb74812f178653 UNKNOWN * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5525152464 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) * cd15b3f75aea1b1598230c8380cb2aa7aa2223af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16801) * 4f9886f810fc14798e4d51b8cadb74812f178653 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5525903403 ## CI report: * cd15b3f75aea1b1598230c8380cb2aa7aa2223af Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16801) * 4f9886f810fc14798e4d51b8cadb74812f178653 UNKNOWN * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5525230063 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) * cd15b3f75aea1b1598230c8380cb2aa7aa2223af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16801) * 4f9886f810fc14798e4d51b8cadb74812f178653 UNKNOWN * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5525381089 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) * cd15b3f75aea1b1598230c8380cb2aa7aa2223af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16801) * 4f9886f810fc14798e4d51b8cadb74812f178653 UNKNOWN * 875600c4ae685f7bd2ec126aafd4d5bc83c96f7e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16804) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923978571
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
+.filter(f -> !(f.schema().isNullable() || f.schema().getType() ==
HoodieSchemaType.NULL))
+.map(HoodieSchemaField::name)
+.collect(Collectors.toList());
+if (requiredCols.isEmpty()) {
+ return schema;
+}
+
+InternalSchema internalSchema = InternalSchemaConverter.convert(schema);
Review Comment:
Kept as on master: the conversion is unchanged here and #19833 tracks the
fix (sentinel ids for the reference fields plus tolerance for the ids already
persisted by schema-on-read tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923992509
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaCompatibility.java:
##
@@ -701,6 +703,120 @@ public void testIsSchemaCompatibleWithTypePromotion() {
assertFalse(HoodieSchemaCompatibility.isSchemaCompatible(longS, intS,
true, true));
}
+ /**
+ * Sibling of {@link #testIsSchemaCompatibleWithTypePromotion()} covering
the rest of the reader/writer type
+ * table: the primitive widening cases shared with {@link
HoodieSchemaTypePromotion}, plus the two
+ * logical-type-over-primitive rules (TIMESTAMP over LONG, UUID over STRING)
that are compatibility-only.
+ *
+ * All pairs are asserted through the 4-arg {@code
isSchemaCompatible(prev = writer, new = reader, true, true)}.
+ */
+ @Test
+ public void testIsSchemaCompatibleWithLogicalTypesAndWidening() {
+// Logical type over its backing primitive: accepted for reader/writer
compatibility.
+assertCompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertCompatible(HoodieSchema.createUUID(),
HoodieSchema.create(HoodieSchemaType.STRING));
+
+// ... but only in that direction, and only over the matching primitive.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.createTimestampMillis());
+assertIncompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.INT));
+// DATE has no such rule at all, even though it is backed by INT.
+assertIncompatible(HoodieSchema.createDate(),
HoodieSchema.create(HoodieSchemaType.INT));
+
+// Primitive widening, delegated to HoodieSchemaTypePromotion.
+assertCompatible(HoodieSchema.create(HoodieSchemaType.DOUBLE),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.FLOAT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.BYTES));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.BYTES),
HoodieSchema.create(HoodieSchemaType.STRING));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.INT));
+ }
+
+ @Test
+ public void testAreSchemasCompatibleReaderIsFirstArgument() {
+HoodieSchema longRecord =
singleFieldRecord(HoodieSchema.create(HoodieSchemaType.LONG));
+HoodieSchema intRecord =
singleFieldRecord(HoodieSchema.create(HoodieSchemaType.INT));
+
+// A long reader can read int data ...
+assertTrue(HoodieSchemaCompatibility.areSchemasCompatible(longRecord,
intRecord));
+// ... but not the other way round, which pins the reader as the FIRST
argument.
+assertFalse(HoodieSchemaCompatibility.areSchemasCompatible(intRecord,
longRecord));
+ }
+
+ @Test
+ public void testLookupWriterFieldDirectMatch() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"a\",\"type\":\"int\"}]}");
+
+HoodieSchemaField writerField =
HoodieSchemaCompatibility.lookupWriterField(writerSchema, readerField);
+assertEquals("a", writerField.name());
+ }
+
+ @Test
+ public void testLookupWriterFieldAliasMatch() {
Review Comment:
Added a javadoc on `testLookupWriterFieldAliasMatch` pointing at
`HoodieTable#validateSchema` /
`TestHoodieTableSchemaEvolution#testFieldWithAlias` in 4f9886f810fc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923989192
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaCompatibility.java:
##
@@ -701,6 +703,120 @@ public void testIsSchemaCompatibleWithTypePromotion() {
assertFalse(HoodieSchemaCompatibility.isSchemaCompatible(longS, intS,
true, true));
}
+ /**
+ * Sibling of {@link #testIsSchemaCompatibleWithTypePromotion()} covering
the rest of the reader/writer type
+ * table: the primitive widening cases shared with {@link
HoodieSchemaTypePromotion}, plus the two
+ * logical-type-over-primitive rules (TIMESTAMP over LONG, UUID over STRING)
that are compatibility-only.
+ *
+ * All pairs are asserted through the 4-arg {@code
isSchemaCompatible(prev = writer, new = reader, true, true)}.
+ */
+ @Test
+ public void testIsSchemaCompatibleWithLogicalTypesAndWidening() {
+// Logical type over its backing primitive: accepted for reader/writer
compatibility.
+assertCompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertCompatible(HoodieSchema.createUUID(),
HoodieSchema.create(HoodieSchemaType.STRING));
+
+// ... but only in that direction, and only over the matching primitive.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.createTimestampMillis());
+assertIncompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.INT));
+// DATE has no such rule at all, even though it is backed by INT.
+assertIncompatible(HoodieSchema.createDate(),
HoodieSchema.create(HoodieSchemaType.INT));
+
+// Primitive widening, delegated to HoodieSchemaTypePromotion.
+assertCompatible(HoodieSchema.create(HoodieSchemaType.DOUBLE),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.FLOAT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.BYTES));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.BYTES),
HoodieSchema.create(HoodieSchemaType.STRING));
+assertCompatible(HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.INT));
+ }
+
+ @Test
+ public void testAreSchemasCompatibleReaderIsFirstArgument() {
+HoodieSchema longRecord =
singleFieldRecord(HoodieSchema.create(HoodieSchemaType.LONG));
+HoodieSchema intRecord =
singleFieldRecord(HoodieSchema.create(HoodieSchemaType.INT));
+
+// A long reader can read int data ...
+assertTrue(HoodieSchemaCompatibility.areSchemasCompatible(longRecord,
intRecord));
+// ... but not the other way round, which pins the reader as the FIRST
argument.
+assertFalse(HoodieSchemaCompatibility.areSchemasCompatible(intRecord,
longRecord));
+ }
+
+ @Test
+ public void testLookupWriterFieldDirectMatch() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"a\",\"type\":\"int\"}]}");
+
+HoodieSchemaField writerField =
HoodieSchemaCompatibility.lookupWriterField(writerSchema, readerField);
+assertEquals("a", writerField.name());
+ }
+
+ @Test
+ public void testLookupWriterFieldAliasMatch() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"old_a\",\"type\":\"int\"}]}");
+
+HoodieSchemaField writerField =
HoodieSchemaCompatibility.lookupWriterField(writerSchema, readerField);
+assertEquals("old_a", writerField.name());
+ }
+
+ @Test
+ public void testLookupWriterFieldAmbiguousMatchThrows() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"a\",\"type\":\"int\"},"
++ "{\"name\":\"old_a\",\"type\":\"int\"}]}");
+
+assertThrows(HoodieSchemaException.class,
+() -> HoodieSchemaCompatibility.lookupWriterField(writerSchema,
readerField));
+ }
+
+ @Test
+ public void testLookupWriterFieldNoMatchReturnsNull() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema writerSchema =
HoodieSchema.parse("{\"type\":\"record\",\"name\":\"W\",\"fields\":["
++ "{\"name\":\"unrelated\",\"type\":\"int\"}]}");
+
+assertNull(HoodieSchemaCompatibility.lookupWriterField(writerSchema,
readerField));
+ }
+
+ @Test
+ public void testLookupWriterFieldRejectsNonRecordWriterSchema() {
+HoodieSchemaField readerField = readerFieldWithAlias();
+HoodieSchema notARecord = HoodieSchema.create(HoodieSchemaType.STRING);
+
+assertThrows(IllegalArgumentException.class,
+() ->
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923987354
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaTypePromotion.java:
##
@@ -19,7 +19,8 @@
package org.apache.hudi.common.schema;
/**
- * The single table of primitive widening promotions, used by {@link
HoodieSchemaProjectionChecker}.
+ * The single table of primitive widening promotions, used by {@link
HoodieSchemaProjectionChecker} and, for the
+ * primitive cases, by {@link HoodieSchemaCompatibilityChecker}.
Review Comment:
Dropped in 86382cd6e3a3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923982053
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaUtils.java:
##
@@ -2084,6 +2086,140 @@ public void
testGetNestedFieldComplexNestedMapAndArray() {
assertEquals(HoodieSchemaType.LONG,
result.get().getRight().schema().getType());
}
+ /**
+ * Record with a namespace, a record-level doc, a custom record prop,
per-field docs and a nested
+ * record - all of its top-level fields required.
+ */
+ private static HoodieSchema allRequiredPersonSchema() {
+HoodieSchema address = HoodieSchema.createRecord(
+"Address",
+"ns.test",
+"the address record",
+Arrays.asList(
+HoodieSchemaField.of("city",
HoodieSchema.create(HoodieSchemaType.STRING), "city doc", null),
+HoodieSchemaField.of("zip",
HoodieSchema.create(HoodieSchemaType.INT), null, null)));
+HoodieSchema schema = HoodieSchema.createRecord(
+"Person",
+"ns.test",
+"the person record",
+Arrays.asList(
+HoodieSchemaField.of("id",
HoodieSchema.create(HoodieSchemaType.INT), "id doc", null),
+HoodieSchemaField.of("name",
HoodieSchema.create(HoodieSchemaType.STRING), null, null),
+HoodieSchemaField.of("address", address, "address doc", null)));
+schema.addProp("hoodie.custom.prop", "custom-value");
+return schema;
+ }
+
+ @Test
+ public void testAsNullableMakesEveryTopLevelFieldNullable() {
Review Comment:
Added `testAsNullablePinsTheInternalSchemaRoundTripLosses` in 4f9886f810fc:
a `default: 0` field comes back with a null default, an ENUM comes back STRING,
a `["string","null"]` union comes back null-first, and a VECTOR(3) column
survives with its dimension. The javadoc names the three losses (86382cd6e3a3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923985484
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
Review Comment:
Done in 86382cd6e3a3: `checkArgument(schema.getType() == RECORD, ...)` after
the null check, `@throws` updated, and `testAsNullableRejectsNonRecordSchema`
pins a STRING and a nullable-record union (4f9886f810fc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923983730
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##
@@ -2552,6 +2554,26 @@ public void testCreateBlob() {
assertFalse(managedOpt.get().schema().isNullable());
}
+ @Test
+ public void testBlobNullableFieldsPutNullFirst() {
Review Comment:
Done in 4f9886f810fc: `testCreateBlob` now ends with
`assertEquals(BLOB_SCHEMA_JSON, blob.toAvroSchema().toString())` against the
literal above, and the null-first loop is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5524935779 @wombatu-kun The second part of the Avro Utils refactoring is ready for review! Can you please help to review it? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923980289
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaCompatibility.java:
##
@@ -701,6 +703,120 @@ public void testIsSchemaCompatibleWithTypePromotion() {
assertFalse(HoodieSchemaCompatibility.isSchemaCompatible(longS, intS,
true, true));
}
+ /**
+ * Sibling of {@link #testIsSchemaCompatibleWithTypePromotion()} covering
the rest of the reader/writer type
+ * table: the primitive widening cases shared with {@link
HoodieSchemaTypePromotion}, plus the two
+ * logical-type-over-primitive rules (TIMESTAMP over LONG, UUID over STRING)
that are compatibility-only.
+ *
+ * All pairs are asserted through the 4-arg {@code
isSchemaCompatible(prev = writer, new = reader, true, true)}.
+ */
+ @Test
+ public void testIsSchemaCompatibleWithLogicalTypesAndWidening() {
+// Logical type over its backing primitive: accepted for reader/writer
compatibility.
+assertCompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.LONG));
+assertCompatible(HoodieSchema.createUUID(),
HoodieSchema.create(HoodieSchemaType.STRING));
+
+// ... but only in that direction, and only over the matching primitive.
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.LONG),
HoodieSchema.createTimestampMillis());
+assertIncompatible(HoodieSchema.createTimestampMillis(),
HoodieSchema.create(HoodieSchemaType.INT));
+// DATE has no such rule at all, even though it is backed by INT.
+assertIncompatible(HoodieSchema.createDate(),
HoodieSchema.create(HoodieSchemaType.INT));
+
+// Primitive widening, delegated to HoodieSchemaTypePromotion.
+assertCompatible(HoodieSchema.create(HoodieSchemaType.DOUBLE),
HoodieSchema.create(HoodieSchemaType.FLOAT));
+assertIncompatible(HoodieSchema.create(HoodieSchemaType.FLOAT),
HoodieSchema.create(HoodieSchemaType.DOUBLE));
Review Comment:
Added in 4f9886f810fc: INT <- LONG/FLOAT/DOUBLE and LONG <- FLOAT/DOUBLE are
asserted incompatible next to the FLOAT <- DOUBLE line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923978571
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
+.filter(f -> !(f.schema().isNullable() || f.schema().getType() ==
HoodieSchemaType.NULL))
+.map(HoodieSchemaField::name)
+.collect(Collectors.toList());
+if (requiredCols.isEmpty()) {
+ return schema;
+}
+
+InternalSchema internalSchema = InternalSchemaConverter.convert(schema);
Review Comment:
Kept as on master: the conversion is unchanged here and #19833 tracks the
fix (sentinel ids for the reference fields plus tolerance for the ids already
persisted by schema-on-read tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923891576
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
+.filter(f -> !(f.schema().isNullable() || f.schema().getType() ==
HoodieSchemaType.NULL))
+.map(HoodieSchemaField::name)
+.collect(Collectors.toList());
+if (requiredCols.isEmpty()) {
+ return schema;
+}
+
+InternalSchema internalSchema = InternalSchemaConverter.convert(schema);
Review Comment:
Filed as #19833 with the repro, will leave this comment unresolved, but will
keep it as a separate fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923833786
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
+.filter(f -> !(f.schema().isNullable() || f.schema().getType() ==
HoodieSchemaType.NULL))
+.map(HoodieSchemaField::name)
+.collect(Collectors.toList());
+if (requiredCols.isEmpty()) {
+ return schema;
+}
+
+InternalSchema internalSchema = InternalSchemaConverter.convert(schema);
Review Comment:
**major:** pre-existing, not introduced by this rewrite: the deleted
`AvroSchemaUtils#asNullable` path gives the identical output.
`InternalSchemaConverter.buildBlobInternalRecordType()` gives the blob's nested
`reference` fields ids 0..3 and `InternalSchema.buildIdToField` is one flat
last-put-wins map, so on `{id: int, b: nullable BLOB}` this call returns a
record whose first field is `external_path` instead of `id`; a required BLOB
throws `Cannot update nullability for column 'b'`. Reachable from Flink
`ClusteringOperator.open()` on any BLOB table. Could we keep the fix out of
this PR so the conversion stays as on master and the PR remains a pure refactor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
voonhous commented on code in PR #19810:
URL: https://github.com/apache/hudi/pull/19810#discussion_r3923833786
##
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##
@@ -248,36 +251,42 @@ public static HoodieSchema mergeSchemas(HoodieSchema
sourceSchema, HoodieSchema
}
/**
- * Creates a nullable version of the given schema (union of null and the
schema).
+ * Create a new schema by force changing all the top-level fields as
nullable.
*
- * {@link HoodieSchema#createNullable(HoodieSchema)} is the idempotent
native equivalent and is
- * preferred; this overload round-trips through Avro and is retained only
for existing call sites.
+ * The rewrite runs through the field-id {@link InternalSchema}: the
record is converted, every
+ * still-required top-level field is marked nullable with a {@link
TableChanges.ColumnUpdateChange},
+ * and the updated InternalSchema is converted back under the original full
name. Only the top level
+ * changes - the inner fields of a nested record keep the nullability they
had. Because the record is
+ * rebuilt from the InternalSchema, its full name, field order and per-field
docs survive, while the
+ * record-level doc and any custom record properties do not.
*
- * @param schema the input schema
- * @return new HoodieSchema that allows null values
- * @throws IllegalArgumentException if schema is null
- */
- public static HoodieSchema createNullableSchema(HoodieSchema schema) {
-ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-
-// Delegate to AvroSchemaUtils
-Schema nullableAvro =
AvroSchemaUtils.createNullableSchema(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
- }
-
- /**
- * Create a new schema by force changing all the fields as nullable.
+ * When every top-level field is already nullable the input instance
itself is returned and no
+ * conversion runs.
*
- * @return a new schema with all the fields updated as nullable
+ * @param schema original schema
+ * @return a schema with all the top-level fields updated as nullable, or
{@code schema} itself when
+ * there is nothing to change
* @throws IllegalArgumentException if schema is null
- * @see AvroSchemaUtils#asNullable(Schema)
*/
public static HoodieSchema asNullable(HoodieSchema schema) {
ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
-// Delegate to AvroSchemaUtils
-Schema nullableAvro = AvroSchemaUtils.asNullable(schema.toAvroSchema());
-return HoodieSchema.fromAvroSchema(nullableAvro);
+// NOTE: HoodieSchema#isNullable is false for a bare NULL type, unlike
Avro's Schema#isNullable, so a
+// NULL-typed field is excluded explicitly to keep it out of the
update list as it always was.
+List requiredCols = schema.getFields().stream()
+.filter(f -> !(f.schema().isNullable() || f.schema().getType() ==
HoodieSchemaType.NULL))
+.map(HoodieSchemaField::name)
+.collect(Collectors.toList());
+if (requiredCols.isEmpty()) {
+ return schema;
+}
+
+InternalSchema internalSchema = InternalSchemaConverter.convert(schema);
Review Comment:
**major:** pre-existing, not introduced by this rewrite: the deleted
`AvroSchemaUtils#asNullable` path gives the identical output.
`InternalSchemaConverter.buildBlobInternalRecordType()` gives the blob's nested
`reference` fields ids 0..3 and `InternalSchema.buildIdToField` is one flat
last-put-wins map, so on `{id: int, b: nullable BLOB}` this call returns a
record whose first field is `external_path` instead of `id`; a required BLOB
throws `Cannot update nullability for column 'b'`. Reachable from Flink
`ClusteringOperator.open()` on any BLOB table. Filed as #19833 with the repro.
Could we keep the fix out of this PR so the conversion stays as on master and
the PR remains a pure refactor?
##
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaUtils.java:
##
@@ -2084,6 +2086,140 @@ public void
testGetNestedFieldComplexNestedMapAndArray() {
assertEquals(HoodieSchemaType.LONG,
result.get().getRight().schema().getType());
}
+ /**
+ * Record with a namespace, a record-level doc, a custom record prop,
per-field docs and a nested
+ * record - all of its top-level fields required.
+ */
+ private static HoodieSchema allRequiredPersonSchema() {
+HoodieSchema address = HoodieSchema.createRecord(
+"Address",
+"ns.test",
+"the address record",
+Arrays.asList(
+HoodieSchemaField.of("city",
HoodieSchema.create(HoodieSchemaType.STRING), "city doc", null),
+HoodieSchemaField.of("zip",
HoodieSchema.create(HoodieSchemaType.INT), null, null)));
+HoodieSchema schema = HoodieSchema.createRecord(
+"Person",
+"ns.test",
+"the person record",
+
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5524599910 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) * cd15b3f75aea1b1598230c8380cb2aa7aa2223af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16801) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5524527302 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) * cd15b3f75aea1b1598230c8380cb2aa7aa2223af UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5524048145 ## CI report: * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5522515828 ## CI report: * a66245aa6387e65564e8d75e7c92cda5865b3c2a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16793) * f8d73749d7e36fd01989fa11537deef3bdc85c38 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5522584652 ## CI report: * a66245aa6387e65564e8d75e7c92cda5865b3c2a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16793) * f8d73749d7e36fd01989fa11537deef3bdc85c38 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16797) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5521839407 ## CI report: * a66245aa6387e65564e8d75e7c92cda5865b3c2a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16793) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5520929640 ## CI report: * 529dc89b21f119b8106432c3058f3ba077fb3ac2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16781) * a66245aa6387e65564e8d75e7c92cda5865b3c2a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16793) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5520881964 ## CI report: * 529dc89b21f119b8106432c3058f3ba077fb3ac2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16781) * a66245aa6387e65564e8d75e7c92cda5865b3c2a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5514980624 ## CI report: * 529dc89b21f119b8106432c3058f3ba077fb3ac2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16781) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5513815379 ## CI report: * 8c1389da004254d7dd8d3fbec0a72c459b896d70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16743) * 529dc89b21f119b8106432c3058f3ba077fb3ac2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16781) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5513727130 ## CI report: * 8c1389da004254d7dd8d3fbec0a72c459b896d70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16743) * 529dc89b21f119b8106432c3058f3ba077fb3ac2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5497175617 ## CI report: * 8c1389da004254d7dd8d3fbec0a72c459b896d70 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16743) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
codecov-commenter commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5496315132 ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19810?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report :x: Patch coverage is `40.0%` with `51 lines` in your changes missing coverage. Please review. :white_check_mark: Project coverage is 78.30%. Comparing base ([`c59987a`](https://app.codecov.io/gh/apache/hudi/commit/c59987a024cd021cbb9aa34bdc74d341ea834b4d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8c1389d`](https://app.codecov.io/gh/apache/hudi/commit/8c1389da004254d7dd8d3fbec0a72c459b896d70?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)). :warning: Report is 1 commits behind head on master. | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19810?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...a/org/apache/hudi/common/avro/HoodieAvroUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Favro%2FHoodieAvroUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2F2cm8vSG9vZGllQXZyb1V0aWxzLmphdmE=) | 26.19% | [25 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...g/apache/hudi/common/schema/HoodieSchemaUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2FHoodieSchemaUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9Ib29kaWVTY2hlbWFVdGlscy5qYXZh) | 40.90% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../hudi/common/schema/HoodieSchemaCompatibility.java](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2FHoodieSchemaCompatibility.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9Ib29kaWVTY2hlbWFDb21wYXRpYmlsaXR5LmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...ommon/schema/HoodieSchemaCompatibilityChecker.java](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2FHoodieSchemaCompatibilityChecker.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9Ib29kaWVTY2hlbWFDb21wYXRpYmlsaXR5Q2hlY2tlci5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [...n/avro/processors/DecimalLogicalTypeProcessor.java](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Favro%2Fprocessors%2FDecimalLogicalTypeProcessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2F2cm8vcHJvY2Vzc29ycy9EZWNpbWFsTG9naWNhbFR5cGVQcm9jZXNzb3IuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5495631252 ## CI report: * 8c1389da004254d7dd8d3fbec0a72c459b896d70 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] refactor(schema): dissolve AvroSchemaUtils, dedupe compat [hudi]
hudi-bot commented on PR #19810: URL: https://github.com/apache/hudi/pull/19810#issuecomment-5495721135 ## CI report: * 8c1389da004254d7dd8d3fbec0a72c459b896d70 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16743) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
