voonhous commented on code in PR #18961:
URL: https://github.com/apache/hudi/pull/18961#discussion_r3850252151
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -298,6 +298,24 @@ public class HoodieStorageConfig extends HoodieConfig {
+ "The provider parses variant binary data and populates typed_value
columns. "
+ "When not set, the provider is auto-detected from the classpath.");
+ public static final ConfigProperty<Boolean>
PARQUET_VARIANT_SHREDDING_SCHEMA_INFERENCE_ENABLED = ConfigProperty
+ .key("hoodie.parquet.variant.shredding.schema.inference.enabled")
+ .defaultValue(false)
+ .sinceVersion("1.3.0")
+ .withDocumentation("When enabled, the shredding schema for variant
columns without an explicit "
+ + "typed_value in the write schema is inferred automatically per
parquet file from a sample of "
+ + "the records written to that file, mirroring Spark 4.1's "
+ + "spark.sql.variant.inferShreddingSchema. Requires Spark 4.1+ on
the writer classpath; "
+ + "writes stay unshredded otherwise (Spark 4.0, Flink, Java
engines). Applies to every "
+ + "parquet file the writer produces: base files and, on table
version 10+, the native "
+ + "parquet log files of MOR tables (each infers its own schema);
legacy Avro log blocks "
Review Comment:
Confirmed: `serializeRecordsToLogBlock` takes the `OutputStream` overload,
which neither factory wraps. The doc now says data blocks inside Avro-format
log files stay unshredded whether Avro or parquet
(`hoodie.logfile.data.block.format`); the PR body sentence is updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
