voonhous opened a new issue, #19825:
URL: https://github.com/apache/hudi/issues/19825
### Bug Description
`HoodieSchemaUtils.hasDecimalField` (`hasDecimalWithCondition` before
#19809) handles a union by recursing on `schema.getNonNullType()`:
```java
case UNION:
return hasDecimalWithCondition(schema.getNonNullType(), condition); //
HoodieSchemaUtils.java:799-800 on master
```
`HoodieSchema.getNonNullType()` returns `this` for a union that has no null
branch (`HoodieSchema.java:1397-1399`), and a union with one null branch and
two or more non-null branches is reduced to exactly that shape
(`HoodieSchema.java:1413`, `createUnion(nonNullTypes)`). So any schema
containing a field typed `["null","string","int"]` (or `["string","int"]`)
recurses until `StackOverflowError`.
Reachable from `SourceFormatAdapter.java:261`, which evaluates
`HoodieSchemaUtils.hasDecimalField(sourceSchema)` for every JSON source
whenever field-name sanitizing is off, i.e. any Hudi Streamer job with a JSON
source whose provided Avro schema carries a multi-branch union.
`HoodieSchemaRepair.hasTimestampMillisField` (`HoodieSchemaRepair.java:249`)
has the identical arm; its caller
`HoodieFileGroupReaderBasedFileFormat.scala:102` receives Spark-derived
schemas, which cannot carry complex unions, so it is latent there.
Present since #17600 (`399af74b369f`) introduced the HoodieSchema version.
Not a regression from #19809, which keeps the arm unchanged.
Minimal reproduction:
```java
HoodieSchema schema = HoodieSchema.createRecord("r", null, null,
Collections.singletonList(
HoodieSchemaField.of("u", HoodieSchema.createUnion(
HoodieSchema.create(HoodieSchemaType.NULL),
HoodieSchema.create(HoodieSchemaType.STRING),
HoodieSchema.create(HoodieSchemaType.INT)))));
HoodieSchemaUtils.hasDecimalField(schema); // StackOverflowError
```
Fix direction: iterate the branches instead of collapsing them, the shape
`HoodieSchema#containsBlobType` (`HoodieSchema.java:1427`) already uses: `case
UNION: return
schema.getTypes().stream().anyMatch(HoodieSchemaUtils::hasDecimalField);`, same
edit in `HoodieSchemaRepair.hasTimestampMillisField`, plus a multi-branch union
case in `TestHoodieSchemaUtils#testHasDecimalField` and
`TestHoodieSchemaRepair`.
### Environment
- Hudi master (`c59987a024cd`), any engine; Hudi Streamer with a JSON source
is the reachable path
### Logs and Stack Trace
```
java.lang.StackOverflowError
at
org.apache.hudi.common.schema.HoodieSchema.getNonNullType(HoodieSchema.java:1391)
at
org.apache.hudi.common.schema.HoodieSchemaUtils.hasDecimalWithCondition(HoodieSchemaUtils.java:800)
at
org.apache.hudi.common.schema.HoodieSchemaUtils.hasDecimalWithCondition(HoodieSchemaUtils.java:800)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
