Fokko commented on code in PR #3608:
URL: https://github.com/apache/parquet-java/pull/3608#discussion_r4402344025


##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -1229,6 +1240,151 @@ public boolean equals(Object obj) {
     }
   }
 
+  /**
+   * File logical type annotation. Annotates a group (struct) that represents 
a reference to a
+   * range of bytes, which may be stored inline in the value or in an external 
file. Every field is
+   * optional, both in the schema (a writer may omit any field from the group 
definition) and in the
+   * data (any field that is present has a field repetition type of {@code 
OPTIONAL}). Fields are
+   * identified by name (case sensitively), not by field order. A group need 
only define the fields
+   * it uses. The group may contain the following fields:
+   * <ul>
+   *   <li>{@code uri} (STRING): a URI-reference (RFC 3986) that identifies an 
external file, for
+   *       example {@code s3://bucket/file.jpg}.</li>
+   *   <li>{@code offset} (INT64): start of the byte range within the external 
file identified by
+   *       {@code uri}; if not set, treated as 0. Must not be negative and may 
only be set together
+   *       with {@code uri}.</li>
+   *   <li>{@code size} (INT64): byte length of the referenced data within the 
external file. May be
+   *       omitted for a whole-file external reference, in which case the 
range runs to the end of
+   *       the referenced file. May only be set together with {@code uri}, 
must be set whenever
+   *       {@code offset} is set, and must not be negative.</li>
+   *   <li>{@code content_type} (STRING): the media (MIME) type (RFC 2046) of 
the resolved bytes;
+   *       when not set, {@code application/octet-stream} is assumed.</li>
+   *   <li>{@code checksum} (STRING): a self-describing integrity token for 
the resolved bytes, of
+   *       the form {@code <algorithm>:<digest>}.</li>
+   *   <li>{@code inline} (BYTE_ARRAY): the referenced bytes stored inline in 
the value.</li>
+   * </ul>
+   * No fields with names other than the above are permitted, and field names 
must be unique. Each
+   * declared field must match its required physical and logical types. Rules 
involving which fields
+   * are set and their values are enforced per value rather than by the schema.
+   */
+  public static class FileLogicalTypeAnnotation extends LogicalTypeAnnotation {
+    private static final FileLogicalTypeAnnotation INSTANCE = new 
FileLogicalTypeAnnotation();
+
+    /** Field name holding the URI-reference of an external file. */
+    public static final String URI_FIELD = "uri";
+
+    /** Field name holding the start of the byte range. */
+    public static final String OFFSET_FIELD = "offset";
+
+    /** Field name holding the byte length of the referenced data. */
+    public static final String SIZE_FIELD = "size";
+
+    /** Field name holding the media (MIME) type of the resolved bytes. */
+    public static final String CONTENT_TYPE_FIELD = "content_type";
+
+    /** Field name holding the integrity token for the resolved bytes. */
+    public static final String CHECKSUM_FIELD = "checksum";
+
+    /** Field name holding the referenced bytes stored inline. */
+    public static final String INLINE_FIELD = "inline";
+
+    /** All recognized field names in a FILE-annotated group. All fields are 
optional. */
+    public static final Set<String> FIELD_NAMES =
+        Set.of(URI_FIELD, OFFSET_FIELD, SIZE_FIELD, CONTENT_TYPE_FIELD, 
CHECKSUM_FIELD, INLINE_FIELD);
+
+    private FileLogicalTypeAnnotation() {}
+
+    /** Validates the fields of a FILE-annotated group. */
+    public void validate(GroupType group) {
+      validateFields(group.getName(), group.getFields());
+    }
+
+    void validateFields(String groupName, List<Type> fields) {

Review Comment:
   This logic feels very similar to what's happening in `FileValueValidator`. 
Maybe we should put it into a single place?



##########
parquet-column/src/main/java/org/apache/parquet/io/api/FileConverters.java:
##########
@@ -0,0 +1,146 @@
+/*

Review Comment:
   More a meta question, should we have a `file` package to store all the 
`File` annotation related logic?



##########
parquet-column/src/main/java/org/apache/parquet/io/api/FileConverters.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.io.api;
+
+import java.util.Objects;
+import java.util.function.Consumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.Type;
+
+/** Converters for assembling values represented by a Parquet {@code 
FILE}-annotated group. */
+public final class FileConverters {
+  private FileConverters() {}
+
+  /**
+   * Creates a converter for a possibly projected FILE group.
+   *
+   * <p>FILE values are atomic and cannot be projected by child field. The 
projected schema may
+   * contain fields absent from this particular file for schema-evolution 
purposes, but it must
+   * contain every FILE field stored in the file schema. The consumer receives 
{@code null} when an
+   * invalid FILE reference is encountered, as permitted by the FILE 
logical-type specification.
+   *
+   * @param schema schema presented to the converter
+   * @param fileSchema FILE group stored in the file
+   * @param consumer consumer for converted values
+   * @return a converter for the complete FILE value
+   * @throws IllegalArgumentException if the projected schema omits a stored 
FILE field
+   */
+  public static GroupConverter newFileConverter(
+      GroupType schema, GroupType fileSchema, Consumer<FileValue> consumer) {
+    Objects.requireNonNull(schema, "schema cannot be null");
+    Objects.requireNonNull(fileSchema, "fileSchema cannot be null");
+    Objects.requireNonNull(consumer, "consumer cannot be null");
+    FileValueValidator.validateSchema(schema);
+    FileValueValidator.validateSchema(fileSchema);
+    for (Type field : fileSchema.getFields()) {
+      if (!schema.containsField(field.getName())) {
+        throw new IllegalArgumentException(
+            "FILE group '" + schema.getName() + "' cannot project stored field 
'" + field.getName() + "'");
+      }
+    }
+    return new FileGroupConverter(schema, consumer);
+  }
+
+  private static final class FileGroupConverter extends GroupConverter {
+    private final Converter[] converters;
+    private final Consumer<FileValue> consumer;
+    private FileValue.Builder builder;
+
+    private FileGroupConverter(GroupType schema, Consumer<FileValue> consumer) 
{
+      this.converters = new Converter[schema.getFieldCount()];
+      this.consumer = consumer;
+      for (int index = 0; index < schema.getFieldCount(); index++) {
+        String fieldName = schema.getFieldName(index);
+        switch (fieldName) {
+          case LogicalTypeAnnotation.FileLogicalTypeAnnotation.URI_FIELD:
+            converters[index] = stringConverter(value -> 
builder.withUri(value));
+            break;
+          case LogicalTypeAnnotation.FileLogicalTypeAnnotation.OFFSET_FIELD:
+            converters[index] = longConverter(value -> 
builder.withOffset(value));
+            break;
+          case LogicalTypeAnnotation.FileLogicalTypeAnnotation.SIZE_FIELD:
+            converters[index] = longConverter(value -> 
builder.withSize(value));
+            break;
+          case 
LogicalTypeAnnotation.FileLogicalTypeAnnotation.CONTENT_TYPE_FIELD:
+            converters[index] = stringConverter(value -> 
builder.withContentType(value));
+            break;
+          case LogicalTypeAnnotation.FileLogicalTypeAnnotation.CHECKSUM_FIELD:
+            converters[index] = stringConverter(value -> 
builder.withChecksum(value));
+            break;
+          case LogicalTypeAnnotation.FileLogicalTypeAnnotation.INLINE_FIELD:
+            converters[index] = binaryConverter(value -> 
builder.withInline(value));
+            break;
+          default:
+            throw new IllegalArgumentException("Unrecognized FILE field: " + 
fieldName);
+        }
+      }
+    }
+
+    @Override
+    public Converter getConverter(int fieldIndex) {
+      return converters[fieldIndex];
+    }
+
+    @Override
+    public void start() {
+      builder = FileValue.builder();
+    }
+
+    @Override
+    public void end() {
+      FileValue value = builder.build();
+      boolean valid = true;
+      try {
+        FileValueValidator.validateValue(value);
+      } catch (IllegalArgumentException e) {
+        valid = false;
+      }
+      builder = null;
+      consumer.accept(valid ? value : null);
+    }
+  }
+
+  private static PrimitiveConverter stringConverter(Consumer<String> consumer) 
{
+    return new PrimitiveConverter() {
+      @Override
+      public void addBinary(Binary value) {
+        consumer.accept(value.toStringUsingUTF8());
+      }
+    };
+  }
+
+  private static PrimitiveConverter binaryConverter(Consumer<Binary> consumer) 
{
+    return new PrimitiveConverter() {
+      @Override
+      public void addBinary(Binary value) {
+        consumer.accept(Binary.fromConstantByteArray(value.getBytes()));

Review Comment:
   This potentially avoids a copy:
   ```suggestion
           consumer.accept(value.copy());
   ```



##########
parquet-column/src/main/java/org/apache/parquet/io/api/FileConverters.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.io.api;
+
+import java.util.Objects;
+import java.util.function.Consumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.Type;
+
+/** Converters for assembling values represented by a Parquet {@code 
FILE}-annotated group. */
+public final class FileConverters {
+  private FileConverters() {}
+
+  /**
+   * Creates a converter for a possibly projected FILE group.
+   *
+   * <p>FILE values are atomic and cannot be projected by child field. The 
projected schema may
+   * contain fields absent from this particular file for schema-evolution 
purposes, but it must
+   * contain every FILE field stored in the file schema. The consumer receives 
{@code null} when an
+   * invalid FILE reference is encountered, as permitted by the FILE 
logical-type specification.
+   *
+   * @param schema schema presented to the converter
+   * @param fileSchema FILE group stored in the file
+   * @param consumer consumer for converted values
+   * @return a converter for the complete FILE value
+   * @throws IllegalArgumentException if the projected schema omits a stored 
FILE field
+   */
+  public static GroupConverter newFileConverter(

Review Comment:
   We have logic here asserting the names, which is great. But it leaves out 
field-IDs completely. Maybe we should check that the fields inside of the FILE 
group don't have any IDs assigned to them



##########
parquet-hadoop/src/test/java/org/apache/parquet/io/api/TestFileValueReadWrite.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.io.api;
+
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.conf.ParquetConfiguration;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.ReadSupport;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Types;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+public class TestFileValueReadWrite extends DirectWriterTest {
+  private static final GroupType FILE_SCHEMA = Types.optionalGroup()
+      .as(LogicalTypeAnnotation.fileType())
+      .optional(BINARY)
+      .as(LogicalTypeAnnotation.stringType())
+      .named("uri")
+      .optional(INT64)
+      .named("offset")
+      .optional(INT64)
+      .named("size")
+      .optional(BINARY)
+      .as(LogicalTypeAnnotation.stringType())
+      .named("content_type")
+      .optional(BINARY)
+      .as(LogicalTypeAnnotation.stringType())
+      .named("checksum")
+      .optional(BINARY)
+      .named("inline")
+      .named("file");
+  private static final MessageType SCHEMA = new MessageType("message", 
FILE_SCHEMA);
+
+  @TempDir
+  private java.nio.file.Path tempDir;
+
+  @Test
+  public void testFileValueRoundTrip() throws IOException {
+    FileValue[] expectedValues = {
+      FileValue.builder()
+          .withUri("s3://bucket/file")
+          .withOffset(10)
+          .withSize(20)
+          .withContentType("image/png")
+          .withChecksum("ETAG:abc")
+          .withInline(Binary.fromString("bytes"))
+          .build(),
+      FileValue.builder().withInline(Binary.fromString("inline only")).build(),
+      null,
+      FileValue.builder().withUri("relative/path").build(),
+      FileValue.builder().withUri("file").withOffset(0).withSize(0).build(),
+      FileValue.builder()
+          .withInline(Binary.fromConstantByteArray(new byte[0]))
+          .build()
+    };
+    Path path = new Path(tempDir.resolve("files.parquet").toUri());
+    try (ParquetWriter<FileValue> writer = new ParquetWriter<>(path, new 
FileWriteSupport())) {
+      for (FileValue value : expectedValues) {
+        writer.write(value);
+      }
+    }
+
+    try (ParquetReader<FileRecord> reader =
+        ParquetReader.builder(new FileReadSupport(), path).build()) {
+      for (FileValue expected : expectedValues) {
+        FileRecord record = reader.read();
+        assertThat(record).isNotNull();
+        if (expected == null) {
+          assertThat(record.value).isNull();
+        } else {
+          FileValue actual = record.value;
+          assertThat(actual).isNotNull();
+          assertThat(actual.getUri()).isEqualTo(expected.getUri());
+          assertThat(actual.getOffset()).isEqualTo(expected.getOffset());
+          assertThat(actual.getSize()).isEqualTo(expected.getSize());
+          
assertThat(actual.getContentType()).isEqualTo(expected.getContentType());
+          assertThat(actual.getChecksum()).isEqualTo(expected.getChecksum());
+          assertThat(actual.getInline()).isEqualTo(expected.getInline());
+        }
+      }
+      assertThat(reader.read()).isNull();
+    }
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = {true, false})
+  public void testInvalidStoredFileValueConvertsToNull(boolean sizeWithoutUri) 
throws IOException {
+    Path path = writeDirect(SCHEMA, consumer -> {
+      consumer.startMessage();
+      consumer.startField("file", 0);
+      consumer.startGroup();
+      if (sizeWithoutUri) {
+        consumer.startField("size", FILE_SCHEMA.getFieldIndex("size"));
+        consumer.addLong(2);
+        consumer.endField("size", FILE_SCHEMA.getFieldIndex("size"));
+      }
+      consumer.startField("content_type", 
FILE_SCHEMA.getFieldIndex("content_type"));
+      consumer.addBinary(Binary.fromString("image/png"));
+      consumer.endField("content_type", 
FILE_SCHEMA.getFieldIndex("content_type"));
+      if (sizeWithoutUri) {
+        consumer.startField("inline", FILE_SCHEMA.getFieldIndex("inline"));
+        consumer.addBinary(Binary.fromString("bytes"));
+        consumer.endField("inline", FILE_SCHEMA.getFieldIndex("inline"));
+      }
+      consumer.endGroup();
+      consumer.endField("file", 0);
+      consumer.endMessage();
+    });
+
+    try (ParquetReader<FileRecord> reader =
+        ParquetReader.builder(new FileReadSupport(), path).build()) {
+      assertThat(reader.read().value).isNull();
+      assertThat(reader.read()).isNull();
+    }
+  }
+
+  @Test
+  public void testFileConverterRejectsPartialProjectionDuringReaderAssembly() 
throws IOException {
+    Path path = writeDirect(SCHEMA, consumer -> {
+      consumer.startMessage();
+      consumer.startField("file", 0);
+      FileValueWriter.write(
+          consumer, FILE_SCHEMA, FileValue.builder().withUri("file").build());
+      consumer.endField("file", 0);
+      consumer.endMessage();
+    });
+    MessageType requestedSchema = new MessageType("message", 
FILE_SCHEMA.withNewFields(FILE_SCHEMA.getType("uri")));
+
+    try (ParquetReader<FileRecord> reader = ParquetReader.builder(new 
FileReadSupport(requestedSchema), path)
+        .build()) {
+      assertThatThrownBy(reader::read)
+          .isInstanceOf(IllegalArgumentException.class)
+          .hasMessage("FILE group 'file' cannot project stored field 
'offset'");
+    }
+  }
+
+  @Test
+  public void testFileConverterReadsReorderedSchemaWithAddedFields() throws 
IOException {
+    GroupType storedFile = 
FILE_SCHEMA.withNewFields(FILE_SCHEMA.getType("uri"));
+    Path path = writeDirect(new MessageType("message", storedFile), consumer 
-> {
+      consumer.startMessage();
+      consumer.startField("file", 0);
+      FileValueWriter.write(
+          consumer, storedFile, FileValue.builder().withUri("file").build());
+      consumer.endField("file", 0);
+      consumer.endMessage();
+    });
+    MessageType requestedSchema = new MessageType(
+        "message", FILE_SCHEMA.withNewFields(FILE_SCHEMA.getType("inline"), 
FILE_SCHEMA.getType("uri")));
+
+    try (ParquetReader<FileRecord> reader = ParquetReader.builder(new 
FileReadSupport(requestedSchema), path)
+        .build()) {
+      FileValue actual = reader.read().value;
+      assertThat(actual.getUri()).isEqualTo("file");
+      assertThat(actual.getInline()).isNull();
+      assertThat(reader.read()).isNull();
+    }
+  }
+
+  private static final class FileRecord {
+    private final FileValue value;
+
+    private FileRecord(FileValue value) {
+      this.value = value;
+    }
+  }
+
+  private static final class FileReadSupport extends ReadSupport<FileRecord> {
+    private final MessageType requestedSchema;
+
+    private FileReadSupport() {
+      this(null);
+    }
+
+    private FileReadSupport(MessageType requestedSchema) {
+      this.requestedSchema = requestedSchema;
+    }
+
+    @Override
+    public ReadContext init(
+        ParquetConfiguration configuration, Map<String, String> metadata, 
MessageType fileSchema) {
+      return new ReadContext(requestedSchema == null ? fileSchema : 
requestedSchema);
+    }
+
+    @Override
+    public RecordMaterializer<FileRecord> prepareForRead(
+        Configuration configuration,
+        Map<String, String> metadata,
+        MessageType fileSchema,
+        ReadContext readContext) {
+      return new FileRecordMaterializer(fileSchema, 
readContext.getRequestedSchema());
+    }
+  }
+
+  private static final class FileWriteSupport extends WriteSupport<FileValue> {
+    private RecordConsumer consumer;
+
+    @Override
+    public WriteContext init(Configuration configuration) {
+      return new WriteContext(SCHEMA, Collections.emptyMap());
+    }
+
+    @Override
+    public void prepareForWrite(RecordConsumer recordConsumer) {
+      consumer = recordConsumer;
+    }
+
+    @Override
+    public void write(FileValue value) {
+      consumer.startMessage();
+      if (value != null) {
+        consumer.startField("file", 0);
+        FileValueWriter.write(consumer, FILE_SCHEMA, value);

Review Comment:
   Within this method, we validate the schema each time, which is pretty 
costly. I don't think we need to do that for `FILE_SCHEMA` on runtime. Ideally, 
we want to validate this before we initiate the write, when constructing the 
`GroupType` with the annotation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to