PDGGK opened a new pull request, #20506:
URL: https://github.com/apache/hudi/pull/20506
### Describe the issue this Pull Request addresses
Closes #19448
With `hoodie.spark.sql.optimized.writes.enable` on (the default), Spark SQL
`UPDATE` and `DELETE` run as prepped writes. Each record's key, partition path
and file group are read from the `_hoodie_record_key`,
`_hoodie_partition_path`, `_hoodie_commit_time` and `_hoodie_file_name` columns
that come back with the scanned rows. A table created with
`hoodie.populate.meta.fields = 'false'` does not store those columns. On
master, a table using one of the selective `hoodie.meta.fields.mode` values
(`COMMIT_TIME_ONLY`, `FILE_NAME_ONLY`, `COMMIT_TIME_AND_FILE_NAME`) does not
store them either. In both cases the statement fails.
The statements from #19448 on 1.2.1:
```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :0
org.apache.hudi.exception.HoodieAppendException: Failed while appending
records to file:<tmp>/t/._20261102161458764.log.1_0-40-60
org.apache.hudi.exception.InvalidHoodiePathException: Invalid path
file:<tmp>/t/._20261102161458764.log.1_0-40-60 of type LogFile
```
The fileId segment of that log name is empty. The same statements on current
master fail earlier, while the records are built:
```
java.lang.NullPointerException: Cannot invoke "Object.toString()" because
the return value of "org.apache.avro.generic.GenericRecord.get(String)" is null
at
org.apache.hudi.HoodieCreateRecordUtils$.getHoodieKeyAndMaybeLocationFromAvroRecord(HoodieCreateRecordUtils.scala:214)
```
With the Spark record merger, the NPE is in
`getHoodieKeyAndMayBeLocationFromSparkRecord`
(`HoodieCreateRecordUtils.scala:254`) instead. `DELETE` fails with the same
NPE. The only workaround today is `set
hoodie.spark.sql.optimized.writes.enable=false`.
### Summary and Changelog
SQL `UPDATE` and `DELETE` now take the prepped path only when the table
stores the record key meta field, that is, when its meta-fields mode is `ALL`.
Other tables use the regular write path, the one
`hoodie.spark.sql.optimized.writes.enable=false` already selects: keys come
from the key generator and locations come from the index.
- `ProvidesHoodieConfig.shouldUsePreppedSqlWrites(sparkSession,
hoodieCatalogTable)`: new helper. It is true when optimized SQL writes are
enabled and `tableConfig.getMetaFieldsMode.isRecordKeyPopulated`.
- `UpdateHoodieTableCommand` and `DeleteHoodieTableCommand`, in both the
spark3-common and spark4-common copies, call the helper in place of the raw
config check. Each command uses it for both of its decisions: the prepped
config flag, and whether the input plan keeps the meta columns.
- The check uses `isRecordKeyPopulated` rather than `populateMetaFields()`,
because the prepped path needs `_hoodie_record_key` and
`_hoodie_partition_path`, which only `ALL` stores. It therefore stays correct
even if the legacy boolean mapping for the selective modes changes.
- New tests: `TestUpdateTable` "Test Update Table Without Meta Fields" and
`TestDeleteTable` "Test Delete Table Without Meta Fields". Each covers COW and
MOR with `hoodie.populate.meta.fields = 'false'`, plus COW with
`hoodie.meta.fields.mode = 'COMMIT_TIME_AND_FILE_NAME'`, under both the Avro
and Spark record mergers.
On this branch, the statements from #19448 succeed with both the Avro and
the Spark record merger. Afterwards, `SELECT id, item.name, ts, part FROM t`
returns `[1,a1,123457,p1]`.
### Impact
`UPDATE` and `DELETE` work on tables that do not populate the record key
meta field. Tables with all meta fields populated, which includes every table
created with default settings, still use the prepped path, so their behavior is
unchanged. No public API or config changes.
### Risk Level
low
The change only affects tables that could not complete a prepped `UPDATE` or
`DELETE` before. Those tables now take the existing non-prepped path, which the
current suites already cover through
`hoodie.spark.sql.optimized.writes.enable=false`. Primary-key-less tables are
not affected: they always populate the record key, because auto-generated keys
reject modes that leave it unpopulated.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
### Testing
Run locally with the Spark 3.5 / Scala 2.12 profile on JDK 17:
- `TestUpdateTable`: 12 tests pass. Of the 12, 11 are existing tests and 1
is new.
- `TestDeleteTable`: 8 tests pass, 7 existing and 1 new.
- `TestDeleteFromTable`: 3 tests pass.
- `TestNestedSchemaPruningOptimization`: 10 tests pass. This suite runs SQL
`UPDATE` with meta fields enabled.
- The two new tests fail on master without the change in this PR, with the
`NullPointerException` above at `HoodieCreateRecordUtils.scala:214`.
- scalastyle reports 0 errors for hudi-spark-common, hudi-spark3-common,
hudi-spark3.5.x and hudi-spark, and for the two spark4-common files.
Left to CI: the Spark 4 profiles. The spark4-common edits are the same lines
as the spark3-common ones, and I did not compile them locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
