exceptionfactory commented on code in PR #10850:
URL: https://github.com/apache/nifi/pull/10850#discussion_r2772355343
##########
nifi-extension-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/test/java/org/apache/nifi/parquet/ParquetTestUtils.java:
##########
@@ -89,5 +94,127 @@ private static ParquetWriter<GenericRecord>
createParquetWriter(final Schema sch
.build();
}
+ /**
+ * Creates a parquet file with timestamp_millis logical type for testing
NIFI-15548.
+ *
+ * @param numRecords Number of records to create
+ * @param directory Directory where the parquet file will be created
+ * @return The created parquet file
+ * @throws IOException if file creation fails
+ */
+ public static File createTimestampParquetFile(int numRecords, File
directory) throws IOException {
Review Comment:
It seems like this case is covered in the all temporal types test, is that
correct?
##########
nifi-extension-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/test/java/org/apache/nifi/parquet/ParquetTestUtils.java:
##########
@@ -89,5 +94,127 @@ private static ParquetWriter<GenericRecord>
createParquetWriter(final Schema sch
.build();
}
+ /**
+ * Creates a parquet file with timestamp_millis logical type for testing
NIFI-15548.
+ *
+ * @param numRecords Number of records to create
+ * @param directory Directory where the parquet file will be created
+ * @return The created parquet file
+ * @throws IOException if file creation fails
+ */
+ public static File createTimestampParquetFile(int numRecords, File
directory) throws IOException {
+ // Create schema with timestamp_millis logical type
+ final Schema timestampSchema =
LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
+
+ final Schema recordSchema = Schema.createRecord("TimestampRecord",
null, "test", false);
+ recordSchema.setFields(List.of(
+ new Schema.Field("id", Schema.create(Schema.Type.INT)),
+ new Schema.Field("name", Schema.create(Schema.Type.STRING)),
+ new Schema.Field("created_at", timestampSchema)
+ ));
+
+ final File parquetFile = new File(directory,
"TestParquetReader-testTimestamp-" + System.currentTimeMillis());
+
+ try (final ParquetWriter<GenericRecord> writer =
createParquetWriter(recordSchema, parquetFile)) {
+ for (int i = 0; i < numRecords; i++) {
+ final GenericRecord record = new
GenericData.Record(recordSchema);
+ record.put("id", i);
+ record.put("name", "Record" + i);
+ // Use epoch millis for timestamp
+ record.put("created_at", Instant.now().toEpochMilli());
+ writer.write(record);
+ }
+ }
+
+ return parquetFile;
+ }
+
+ /**
+ * Creates a parquet file with nullable timestamp_millis logical type for
testing NIFI-15548.
+ * The timestamp field is wrapped in a union with null.
+ *
+ * @param numRecords Number of records to create
+ * @param directory Directory where the parquet file will be created
+ * @return The created parquet file
+ * @throws IOException if file creation fails
+ */
+ public static File createNullableTimestampParquetFile(int numRecords, File
directory) throws IOException {
+ // Create schema with nullable timestamp_millis logical type (union
with null)
+ final Schema timestampSchema =
LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
+ final Schema nullableTimestampSchema =
Schema.createUnion(Schema.create(Schema.Type.NULL), timestampSchema);
Review Comment:
Instead of having an entire method, what about combing this nullable field
with the other fields in the all temporal types test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
