voonhous commented on code in PR #19855:
URL: https://github.com/apache/hudi/pull/19855#discussion_r3965112036
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -520,6 +520,41 @@ public static HoodieSchema
generateProjectionSchema(HoodieSchema originalSchema,
return HoodieSchema.createRecord(originalSchema.getName(),
originalSchema.getNamespace().orElse(null),
originalSchema.getDoc().orElse(null), projectedFields);
}
+ /**
+ * Generate a reader schema off the provided writeSchema, to just project
out the provided columns.
+ *
+ * <p>This overload is intended for callers that already have a
name-to-field map,
+ * such as the realtime reader.</p>
+ *
+ * @param writeSchema the source schema
+ * @param schemaFieldsMap prebuilt case-insensitive field-name map
+ * @param fieldNames the list of field names to include in the
projection
+ * @param isError whether the projected schema is an error schema
+ * @return new HoodieSchema containing only the specified fields
+ */
+ public static HoodieSchema generateProjectionSchema(HoodieSchema writeSchema,
+ Map<String,
HoodieSchemaField> schemaFieldsMap,
+ List<String> fieldNames,
+ boolean isError) {
+ ValidationUtils.checkArgument(writeSchema != null, "Write schema cannot be
null");
+ ValidationUtils.checkArgument(schemaFieldsMap != null, "Schema fields map
cannot be null");
Review Comment:
Addressed in e056b035 -- the Avro/Presto vs Hive/Impala/SparkSQL
case-sensitivity note is back, on the map-taking overload.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java:
##########
@@ -183,7 +183,7 @@ private void init() throws Exception {
// TODO(vc): In the future, the reader schema should be updated based on
log files & be able
// to null out fields not present before
- readerSchema =
HoodieRealtimeRecordReaderUtils.generateProjectionSchema(writerSchema,
schemaFieldsMap, projectionFields);
+ readerSchema = HoodieSchemaUtils.generateProjectionSchema(writerSchema,
schemaFieldsMap, projectionFields,writerSchema.isError());
Review Comment:
Gone in ebaa63ff -- the `isError` argument was dropped, so the call site is
now `generateProjectionSchema(writerSchema, schemaFieldsMap, projectionFields)`.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -520,6 +520,51 @@ public static HoodieSchema
generateProjectionSchema(HoodieSchema originalSchema,
return HoodieSchema.createRecord(originalSchema.getName(),
originalSchema.getNamespace().orElse(null),
originalSchema.getDoc().orElse(null), projectedFields);
}
+ /**
+ * Generate a reader schema off the provided writeSchema, to just project
out the provided columns.
+ *
+ * <p>This overload is intended for callers that already have a
name-to-field map,
+ * such as the realtime reader.</p>
+ *
+ * @param writeSchema the source schema
+ * @param schemaFieldsMap prebuilt case-insensitive field-name map
+ * @param fieldNames the list of field names to include in the
projection
+ * @param isError whether the projected schema is an error schema
+ * @return new HoodieSchema containing only the specified fields
+ */
+ public static HoodieSchema generateProjectionSchema(HoodieSchema writeSchema,
+ Map<String,
HoodieSchemaField> schemaFieldsMap,
+ List<String> fieldNames,
+ boolean isError) {
Review Comment:
Addressed in ebaa63ff -- the parameter is gone and the overload reads
`writeSchema.isError()` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
