SEPURI-SAI-KRISHNA commented on code in PR #19648:
URL: https://github.com/apache/hudi/pull/19648#discussion_r3815116952
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java:
##########
@@ -213,4 +213,34 @@ void
testSlashSeparatedDatePartitioningWithAlreadyFormattedInput() {
Assertions.assertEquals("key1", key.getRecordKey());
Assertions.assertEquals("2026/01/01", key.getPartitionPath());
}
+
+ @Test
+ void testSlashSeparatedDatePartitioningOnRowWritingPaths() {
+ TypedProperties properties = getPropsWithSlashSeparatedDatePartitioning();
+ // NOTE: "ts_ms" is the string-typed field of the example schema,
"timestamp" is a long
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"ts_ms");
+ SimpleKeyGenerator keyGenerator = new SimpleKeyGenerator(properties);
+
+ GenericRecord avroRecord = getRecord();
+ Assertions.assertEquals("2020/03/21",
keyGenerator.getPartitionPath(avroRecord));
+
+ Row row = KeyGeneratorTestUtilities.getRow(avroRecord);
+ Assertions.assertEquals("2020/03/21", keyGenerator.getPartitionPath(row));
+
+ InternalRow internalRow = KeyGeneratorTestUtilities.getInternalRow(row);
+ Assertions.assertEquals(UTF8String.fromString("2020/03/21"),
+ keyGenerator.getPartitionPath(internalRow, row.schema()));
+ }
+
+ @Test
+ void testSlashSeparatedDatePartitioningWithNullValue() {
+ TypedProperties properties = getPropsWithSlashSeparatedDatePartitioning();
+ properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(),
"nested_col.prop1");
+ SimpleKeyGenerator keyGenerator = new SimpleKeyGenerator(properties);
+
+ GenericRecord avroRecord = getRecord(getNestedColRecord(null, 10L));
+
+ Row row = KeyGeneratorTestUtilities.getRow(avroRecord);
+ Assertions.assertEquals(HUDI_DEFAULT_PARTITION_PATH,
keyGenerator.getPartitionPath(row));
Review Comment:
Applied, with one adjustment: the suggestion as written cannot run.
KeyGeneratorTestUtilities#getInternalRow builds a flat GenericInternalRow via
InternalRow.apply, so the "nested_col" value stays a GenericRowWithSchema and
reading it back as a struct off the InternalRow fails with
ClassCastException: GenericRowWithSchema cannot be cast to InternalRow
Every other getInternalRow call site in these tests uses a top-level
partition field, so this is the first to point it at a nested one. A null on a
top-level field is not an escape either -- every other field of EXAMPLE_SCHEMA
is non-nullable, and HoodieUnsafeRowUtils rejects the null before the formatter
is reached ("Found null value for the field that is declared as non-nullable:
StructField(ts_ms,StringType,false)").
So the InternalRow is built through Spark's own converter instead, which
handles the nesting:
InternalRow internalRow =
(InternalRow)
CatalystTypeConverters.createToCatalystConverter(row.schema()).apply(row);
Same coverage you asked for, without changing the shared helper (that would
touch 12 unrelated call sites). Verified red on master -- NullPointerException
at the Row assert -- and green here.
Happy to fix getInternalRow properly in a follow-up if you would rather have
the helper handle nesting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
