wombatu-kun commented on code in PR #20506:
URL: https://github.com/apache/hudi/pull/20506#discussion_r4488484607
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -572,6 +572,17 @@ object ProvidesHoodieConfig {
HoodieCommonConfig.SCHEMA_EVOLUTION_ENABLE.defaultValue.toString)
).toBoolean
+ /**
+ * Whether a Spark SQL UPDATE or DELETE on the table is written as a prepped
write, which takes each
+ * record's key, partition path and file group from the meta columns read
along with it. A table that
+ * does not populate those meta columns has nothing to read there, so it
goes through the regular write
+ * path, which derives the keys with the key generator and locates the
records through the index.
+ */
+ def shouldUsePreppedSqlWrites(sparkSession: SparkSession,
hoodieCatalogTable: HoodieCatalogTable): Boolean =
Review Comment:
With `'hoodie.table.keygenerator.class' =
'org.apache.hudi.keygen.ComplexKeyGenerator'` on the COW `populate.meta.fields
= 'false'` case, the non-prepped fallback leaves a duplicate `id = 1` row after
the UPDATE and keeps `id = 2` after the DELETE, where master fails with the NPE
(the key generator emits `id:1`, but `RecordContext` keys the base rows by the
bare value). Could you add that case to both tests and either fix the key
mismatch or fail fast for that shape?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestUpdateTable.scala:
##########
@@ -541,6 +541,46 @@ class TestUpdateTable extends HoodieSparkSqlTestBase {
}
}
+ test("Test Update Table Without Meta Fields") {
Review Comment:
With this fix, the `Meta fields stay enabled ... since the SQL UPDATE on MOR
needs them to locate the file group` notes in
`TestNestedSchemaPruningOptimization` are no longer true. Not a blocker, but
could you drop them and let the partitioned array/map test use the helper's
default `populateMetaFields = false`, since that test is the issue's own repro?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
