voonhous opened a new issue, #19734:
URL: https://github.com/apache/hudi/issues/19734
### Describe the problem
`BaseFileOnlyRelation.buildScan` with zero required columns and
`hoodie.schema.on.read.enable=true` throws `NullPointerException: Cannot invoke
Types$RecordType.fields() because "record" is null` from
`InternalSchema.buildIdToName`.
`HoodieBaseRelation.projectSchema` calls
`InternalSchemaUtils.pruneInternalSchema(schema, [])`; `pruneType` keeps no
field and returns null, and `pruneInternalSchemaByID` wraps it in `new
InternalSchema(null)`. `BaseFileOnlyRelation.mandatoryFields` is empty, so
nothing pads the projection.
Unreachable from batch SQL: `DefaultSource` routes reads to the
file-group-reader relations, whose
`ParquetSchemaEvolutionUtils.pruneInternalSchema` returns the unpruned schema
for an empty projection. Reachable for streaming reads with
`hoodie.file.group.reader.enabled=false` and for metadata-table reads, both
under schema-on-read.
### To Reproduce
Construct `BaseFileOnlyRelation` directly on a schema-on-read table (as
`TestLegacyParquetReadPath` does) and call `.count()`. Surfaced by #19687's
`TestLegacyParquetReadPath.testCowSnapshotReadWithSchemaOnReadRejectsShreddedVariant`,
whose `count()` carve-out was dropped in that PR because of this.
### Expected behavior
`count()` returns the row count, as it does without schema-on-read (the
`Left` branch of `projectSchema` builds an empty record).
Candidate fix: return `InternalSchema.getEmptyInternalSchema` (or an
empty-record `InternalSchema`) from `projectSchema` when `requiredColumns` is
empty, mirroring the non-schema-on-read branch. Then restore the `count()`
assertion in
`TestLegacyParquetReadPath.testCowSnapshotReadWithSchemaOnReadRejectsShreddedVariant`
and add a non-variant `count()` leg on the legacy relation.
### Environment Description
* Hudi version: master (spark4.2 / scala-2.13 CI lane of #19687, head
fab55f1948ca)
* Spark version: 4.2.0
* Storage: local
### Stacktrace
```
java.lang.NullPointerException: Cannot invoke
"org.apache.hudi.common.schema.internal.Types$RecordType.fields()" because
"record" is null
at
org.apache.hudi.common.schema.internal.InternalSchema.buildIdToName(InternalSchema.java:94)
at
org.apache.hudi.common.schema.internal.InternalSchema.<init>(InternalSchema.java:86)
at
org.apache.hudi.common.schema.internal.InternalSchema.<init>(InternalSchema.java:73)
at
org.apache.hudi.common.schema.internal.utils.InternalSchemaUtils.pruneInternalSchemaByID(InternalSchemaUtils.java:98)
at
org.apache.hudi.common.schema.internal.utils.InternalSchemaUtils.pruneInternalSchema(InternalSchemaUtils.java:73)
at
org.apache.hudi.HoodieBaseRelation$.projectSchema(HoodieBaseRelation.scala:753)
at
org.apache.hudi.HoodieBaseRelation.buildScan(HoodieBaseRelation.scala:320)
at
org.apache.spark.sql.execution.datasources.DataSourceStrategy$.$anonfun$apply$4(DataSourceStrategy.scala:418)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
