[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7175: [HUDI-5191] Fix compatibility with avro 1.10

2022-12-13 Thread GitBox 


Zouxxyy commented on code in PR #7175:
URL: https://github.com/apache/hudi/pull/7175#discussion_r1047921632


##
.github/workflows/bot.yml:
##
@@ -73,6 +73,14 @@ jobs:
 run: |
   HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q 
-DforceStdout)
   ./packaging/bundle-validation/ci_run.sh $HUDI_VERSION
+  - name: Common Test

Review Comment:
   @alexeykudinkin Because the patch repairs the avro compatibility of the test 
cases in the hudi-common module under spark3, the test of the hudi-common 
module is added in bot



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7175: [HUDI-5191] Fix compatibility with avro 1.10

2022-11-11 Thread GitBox 


Zouxxyy commented on code in PR #7175:
URL: https://github.com/apache/hudi/pull/7175#discussion_r1020629125


##
hudi-common/src/test/java/org/apache/hudi/common/model/TestPartialUpdateAvroPayload.java:
##
@@ -55,7 +55,7 @@ public class TestPartialUpdateAvroPayload {
   + "{\"name\": \"id\", \"type\": [\"null\", \"string\"]},\n"
   + "{\"name\": \"partition\", \"type\": [\"null\", \"string\"]},\n"
   + "{\"name\": \"ts\", \"type\": [\"null\", \"long\"]},\n"
-  + "{\"name\": \"_hoodie_is_deleted\", \"type\": [\"null\", 
\"boolean\"], \"default\":false},\n"
+  + "{\"name\": \"_hoodie_is_deleted\", \"type\": [\"boolean\", 
\"null\"], \"default\":false},\n"

Review Comment:
   > actually `_hoodie_is_deleted` should never be null. let's make it just 
boolean type?
   
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7175: [HUDI-5191] Fix compatibility with avro 1.10

2022-11-11 Thread GitBox 


Zouxxyy commented on code in PR #7175:
URL: https://github.com/apache/hudi/pull/7175#discussion_r1020046817


##
hudi-common/src/test/java/org/apache/hudi/common/util/TestObjectSizeCalculator.java:
##
@@ -73,7 +74,8 @@ public void testGetObjectSize() {
 assertEquals(32, getObjectSize(emptyClass));
 assertEquals(40, getObjectSize(stringClass));
 assertEquals(40, getObjectSize(payloadClass));
-assertEquals(1240, getObjectSize(Schema.create(Schema.Type.STRING)));
+assertEquals(HoodieAvroUtils.gteqAvro1_10() ? 1320 : 1240,

Review Comment:
   done



##
hudi-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroUtils.java:
##
@@ -253,7 +254,13 @@ public void testRemoveFields() {
 assertEquals("key1", rec1.get("_row_key"));
 assertEquals("val1", rec1.get("non_pii_col"));
 assertEquals(3.5, rec1.get("timestamp"));
-assertNull(rec1.get("pii_col"));
+Object removed;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7175: [HUDI-5191] Fix compatibility with avro 1.10

2022-11-11 Thread GitBox 


Zouxxyy commented on code in PR #7175:
URL: https://github.com/apache/hudi/pull/7175#discussion_r1020046631


##
hudi-common/src/main/java/org/apache/hudi/common/model/debezium/AbstractDebeziumAvroPayload.java:
##
@@ -76,7 +77,12 @@ private Option 
handleDeleteOperation(IndexedRecord insertRecord)
 boolean delete = false;
 if (insertRecord instanceof GenericRecord) {
   GenericRecord record = (GenericRecord) insertRecord;
-  Object value = record.get(DebeziumConstants.FLATTENED_OP_COL_NAME);
+  Object value;
+  try {
+value = record.get(DebeziumConstants.FLATTENED_OP_COL_NAME);
+  } catch (AvroRuntimeException e) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7175: [HUDI-5191] Fix compatibility with avro 1.10

2022-11-10 Thread GitBox 


Zouxxyy commented on code in PR #7175:
URL: https://github.com/apache/hudi/pull/7175#discussion_r1019779340


##
hudi-common/src/test/java/org/apache/hudi/common/model/TestOverwriteNonDefaultsWithLatestAvroPayload.java:
##
@@ -181,10 +181,10 @@ public void testDeletedRecord() throws IOException {
   @Test
   public void testNullColumn() throws IOException {
 Schema avroSchema = Schema.createRecord(Arrays.asList(
-new Schema.Field("id", 
Schema.createUnion(Schema.create(Schema.Type.STRING), 
Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
-new Schema.Field("name", 
Schema.createUnion(Schema.create(Schema.Type.STRING), 
Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
-new Schema.Field("age", 
Schema.createUnion(Schema.create(Schema.Type.STRING), 
Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
-new Schema.Field("job", 
Schema.createUnion(Schema.create(Schema.Type.STRING), 
Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE)
+new Schema.Field("id", 
Schema.createUnion(Schema.create(Schema.Type.NULL), 
Schema.create(Schema.Type.STRING)), "", JsonProperties.NULL_VALUE),

Review Comment:
   After avro 1.10, the defaultValue is checked when the Schema is initialized, 
and here are the 
[rules](https://avro.apache.org/docs/1.11.1/specification/#unions) of the union
   
   > Unions, as mentioned above, are represented using JSON arrays. For 
example, ["null", "string"] declares a schema which may be either a null or 
string.
   > (Note that when a default value is specified for a record field whose type 
is a union, **the type of the default value must match the first element of the 
union**. Thus, for unions containing “null”, the “null” is usually listed 
first, since the default value of such unions is typically null.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org