wmoustafa commented on code in PR #9366:
URL: https://github.com/apache/iceberg/pull/9366#discussion_r1454299740
##########
core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java:
##########
@@ -69,62 +99,108 @@ public void setRowPositionSupplier(Supplier<Long>
posSupplier) {
@Override
public T read(T reuse, Decoder decoder) throws IOException {
- return DecoderResolver.resolveAndRead(decoder, readSchema, fileSchema,
reader, reuse);
+ return reader.read(decoder, reuse);
}
- private static class ReadBuilder extends AvroSchemaVisitor<ValueReader<?>> {
- private final ClassLoader loader;
+ private class ResolvingReadBuilder extends AvroWithPartnerVisitor<Type,
ValueReader<?>> {
+ private final Map<Type, Schema> avroSchemas;
- private ReadBuilder(ClassLoader loader) {
- this.loader = loader;
+ private ResolvingReadBuilder(Types.StructType expectedType, String
rootName) {
+ this.avroSchemas = AvroSchemaUtil.convertTypes(expectedType, rootName);
}
@Override
- @SuppressWarnings("unchecked")
- public ValueReader<?> record(Schema record, List<String> names,
List<ValueReader<?>> fields) {
- try {
- Class<?> recordClass =
-
DynClasses.builder().loader(loader).impl(record.getFullName()).buildChecked();
- if (IndexedRecord.class.isAssignableFrom(recordClass)) {
- return ValueReaders.record(fields, (Class<? extends IndexedRecord>)
recordClass, record);
+ public ValueReader<?> record(Type partner, Schema record,
List<ValueReader<?>> fieldResults) {
+ Types.StructType expected = partner != null ? partner.asStructType() :
null;
+ Map<Integer, Integer> idToPos = idToPos(expected);
+
+ List<Pair<Integer, ValueReader<?>>> readPlan = Lists.newArrayList();
+ List<Schema.Field> fileFields = record.getFields();
+ for (int pos = 0; pos < fileFields.size(); pos += 1) {
+ Schema.Field field = fileFields.get(pos);
+ ValueReader<?> fieldReader = fieldResults.get(pos);
+ Integer fieldId = AvroSchemaUtil.fieldId(field);
+ Integer projectionPos = idToPos.remove(fieldId);
+
+ Object constant = idToConstant.get(fieldId);
+ if (projectionPos != null && constant != null) {
+ readPlan.add(
+ Pair.of(projectionPos,
ValueReaders.replaceWithConstant(fieldReader, constant)));
+ } else {
+ readPlan.add(Pair.of(projectionPos, fieldReader));
+ }
+ }
+
+ // handle any expected columns that are not in the data file
+ for (Map.Entry<Integer, Integer> idAndPos : idToPos.entrySet()) {
+ int fieldId = idAndPos.getKey();
+ int pos = idAndPos.getValue();
+
+ Object constant = idToConstant.get(fieldId);
+ Types.NestedField field = expected.field(fieldId);
+ if (constant != null) {
+ readPlan.add(Pair.of(pos, ValueReaders.constant(constant)));
Review Comment:
I think here we need something along the lines of
`GenericAvroConstantReader` that returns the constant in the Avro
`GenericData.Record` format. Right now the `ConstantReader` class returns the
constant object as is. Most of the time this constant is an Iceberg data
constant, but what we need here is an Avro `GenericData.Record`. We can extend
`ConstantReader<T>` here, but is is a private class. Can we promote it to
public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
