danny0405 commented on code in PR #8840:
URL: https://github.com/apache/hudi/pull/8840#discussion_r1211045326
##########
hudi-common/src/main/java/org/apache/hudi/common/util/JsonUtils.java:
##########
@@ -20,41 +20,74 @@
package org.apache.hudi.common.util;
import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.util.Lazy;
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.util.StdDateFormat;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
/**
* Utils for JSON serialization and deserialization.
*/
public class JsonUtils {
- private static final ObjectMapper MAPPER = new ObjectMapper();
-
- static {
- MAPPER.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
- // We need to exclude custom getters, setters and creators which can use
member fields
- // to derive new fields, so that they are not included in the serialization
- MAPPER.setVisibility(PropertyAccessor.FIELD,
JsonAutoDetect.Visibility.ANY);
- MAPPER.setVisibility(PropertyAccessor.GETTER,
JsonAutoDetect.Visibility.NONE);
- MAPPER.setVisibility(PropertyAccessor.IS_GETTER,
JsonAutoDetect.Visibility.NONE);
- MAPPER.setVisibility(PropertyAccessor.SETTER,
JsonAutoDetect.Visibility.NONE);
- MAPPER.setVisibility(PropertyAccessor.CREATOR,
JsonAutoDetect.Visibility.NONE);
- }
+ private static final Lazy<ObjectMapper> MAPPER =
Lazy.lazily(JsonUtils::instantiateObjectMapper);
public static ObjectMapper getObjectMapper() {
- return MAPPER;
+ return MAPPER.get();
}
public static String toString(Object value) {
try {
- return MAPPER.writeValueAsString(value);
+ return MAPPER.get().writeValueAsString(value);
} catch (JsonProcessingException e) {
throw new HoodieIOException(
"Fail to convert the class: " + value.getClass().getName() + " to
Json String", e);
}
}
+
+ private static ObjectMapper instantiateObjectMapper() {
+ ObjectMapper mapper = new ObjectMapper();
+
+ registerModules(mapper);
+
+ // We're writing out dates as their string representations instead of
(int) timestamps
+ mapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ // NOTE: This is necessary to make sure that w/ Jackson >= 2.11 colon is
not infixed
+ // into the timezone value ("+00:00" as opposed to "+0000" before
2.11)
+ // While Jackson is able to parse both of these formats, we keep it
as false
+ // to make sure metadata produced by Hudi stays consistent across
Jackson versions
+ configureColonInTimezone(mapper);
Review Comment:
Seems the main fix is line 60 ~ line 65.
Wondering why we serialize the data type as date time string instead of
epoch days. Can you clarify what is the scope of the change, because the column
stats uses Avro serialization which store the data as java `LocalDate` already,
curious how the change could affect column stats serialization.
If we remove line 60 ~ 65, test `TestColumnStatsIndex` still passes, so what
exactly are we fixing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
