SEPURI-SAI-KRISHNA opened a new pull request, #19648:
URL: https://github.com/apache/hudi/pull/19648
### Describe the issue this Pull Request addresses
Closes #19647.
With `hoodie.datasource.write.slash.separated.date.partitioning=true`, every
write that goes through
Spark's row writer (bulk insert, `INSERT INTO` with
`hoodie.sql.bulk.insert.enable=true`, and any
path calling `SparkKeyGeneratorInterface#getPartitionPath(InternalRow,
StructType)`) fails with:
```
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String
cannot be cast to
class java.lang.String
```
and a `null` value in the partition column fails with a
`NullPointerException` on both the row
writer and the `Row` write paths.
`PartitionPathFormatterBase#combine` is generic over the string
representation `S` it builds
(`String` for the Avro/`Row` write paths, `UTF8String` for the row writer),
but the
slash-separated-date branch hard-casts to `java.lang.String`:
```java
if (slashSeparatedDatePartitioning) {
return ((S) ((String) toString(partitionPathParts[0])).replace('-', '/'));
} else {
return tryEncode(handleEmpty(toString(partitionPathParts[0])));
}
```
That branch also skips `handleEmpty` and `tryEncode`, so a null/empty
partition value NPEs instead
of landing in `__HIVE_DEFAULT_PARTITION__`, and URL encoding is silently
dropped.
The multi-field loop has the same hard-cast, and additionally applied the
`-` -> `/` substitution to
*every* partition field, which the Avro write path
(`KeyGenUtils#getRecordPartitionPath`) does not do — it only substitutes
when the table is
partitioned by a single column. The two write paths therefore derived
different partition paths for
the same record.
The feature's existing coverage only exercises the Avro key-generator path
(`getKey(GenericRecord)`), which is why none of this was caught.
### Summary and Changelog
Slash-separated date partitioning now works on the Spark row-writer path and
handles null partition
values, and both write paths derive the same partition path.
- `PartitionPathFormatterBase`: replaced the `(String)` hard-cast with a new
abstract
`replaceDashesWithSlashes(S)` hook, so the substitution happens on the
concrete string
representation the formatter operates on.
- `PartitionPathFormatterBase`: the single-field fast path now runs
`handleEmpty` and `tryEncode`
before the substitution, matching the multi-field loop and
`KeyGenUtils#getPartitionPath`.
- `PartitionPathFormatterBase`: dropped the substitution from the
multi-field loop, aligning it with
`KeyGenUtils#getRecordPartitionPath` on the Avro write path
(slash-separated date partitioning
only applies to a table partitioned by a single date column). Hive-style
partitioning keeps
precedence exactly as before.
- `StringPartitionPathFormatter` / `UTF8StringPartitionPathFormatter`:
implement the new hook
(`String#replace` and `UTF8String#replace` respectively).
- Tests: new `TestPartitionPathFormatter` covering both formatter flavors
(single/multi field, null
and empty values, encoding, hive-style precedence); new row-writer and
null-value cases in
`TestSimpleKeyGenerator`; new end-to-end row-writer case in
`TestSlashSeparatedPartitionValue`.
Verified that each new test fails on `master` with the exact
`ClassCastException` /
`NullPointerException` above and passes with the fix.
### Impact
Fixes a hard failure on the Spark row-writer write path for tables using
`hoodie.datasource.write.slash.separated.date.partitioning`. No public API
change and no storage
format change.
Affects 1.2.0 and master: the feature landed in #17787, which is an ancestor
of `release-1.2.0`.
Behavior change: for a table with **more than one** partition field and
slash-separated date
partitioning enabled, the `Row`/`InternalRow` write paths no longer
substitute `-` with `/` in the
partition path. This makes them agree with the Avro write path, which is the
behavior the table was
already written with — the row-writer variant of that combination could not
previously run at all
(it threw the `ClassCastException`).
### Risk Level
low
The change is confined to partition-path formatting for the row-writer paths
of the key generators.
The previously broken paths threw before producing any output, and the newly
aligned multi-field
behavior matches the Avro write path that existing tables were written with.
Covered by 103
`hudi-spark-client` key-generator/bulk-insert tests and the
`TestSlashSeparatedPartitionValue` and
`TestShowPartitions` Spark SQL suites, all green.
### Documentation Update
none — no new configs and no change to the documented behavior of
`hoodie.datasource.write.slash.separated.date.partitioning`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
