Indian-boult opened a new pull request, #20738:
URL: https://github.com/apache/hudi/pull/20738
### Describe the issue this Pull Request addresses
Closes #19766
On a schema-on-read table with at least one commit, `ALTER TABLE ... RENAME
COLUMN` and `ALTER TABLE ... DROP COLUMN` on a top-level column fail under
default configs with `MissingSchemaFieldException`. #13595 added
`HoodieTable.validateSchema()` to `AlterTableCommand.commitWithSchema` for its
secondary index check. `validateSchema` is the write path check: with
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop=false` (the
default) it rejects any table column that is missing from the incoming schema,
and a rename or a drop always produces one.
Two more problems come from the same call:
- It runs after `client.startCommit`, so every rejected ALTER leaves a
REQUESTED instant on the timeline. Once a later ALTER commits after it, the
next write fails while rolling it back: `HoodieRollbackException: Found commits
after time :<instant>, please rollback greater commits first`, and leaves a
rollback INFLIGHT behind. Reproduced on COW and MOR with a rejected rename, a
successful `ADD COLUMNS`, then an `INSERT`.
- With the column drop config set to `true` (and
`hoodie.avro.schema.validate` at its default), `validateSchema` returns before
any check, so the secondary index check of #13595 is skipped too: `ALTER COLUMN
... TYPE` on a column with a secondary index goes through.
### Summary and Changelog
ALTER TABLE is an explicit schema change, so it now runs only the secondary
index check, before the commit starts. Renaming and dropping columns under
default configs works again, a rejected ALTER leaves nothing on the timeline,
and the secondary index check no longer depends on the column drop config.
- `HoodieTable#validateAlterSchema`: new. Runs
`validateSecondaryIndexSchemaEvolution` against the writer schema, with the
same skip conditions as `validateSchema` (no index metadata, no completed
commit, no writer schema), and does not read
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop` or
`hoodie.avro.schema.validate`. `validateSchema` and the write path are
unchanged.
- `HoodieTable#validateSecondaryIndexSchemaEvolution`: new
`allowMissingColumns` argument; the existing 3-argument form passes `true` and
behaves as before. `validateAlterSchema` passes `false`, so renaming or
dropping a column that has a secondary index is rejected with "Column 'q' has
secondary index 'secondary_index_idx_q' and cannot be renamed or dropped".
Under default configs that ALTER was already rejected on master (by the
`MissingSchemaFieldException` above), so this keeps that outcome with an exact
message. Without it, the fix would let the rename through and leave the index
stale: on master with the column drop config on, a renamed indexed column gets
null secondary keys for new rows and keeps the old keys on updates, and the
index definition keeps the old name.
- `AlterTableCommand.commitWithSchema`: calls `validateAlterSchema` before
`client.startCommit` instead of `validateSchema` after it. It is the only ALTER
path that called `validateSchema`: with schema-on-read, ADD COLUMNS, RENAME
COLUMN, DROP COLUMN, REPLACE COLUMNS and ALTER COLUMN on Spark 3.x
(`AlterColumn`) and 4.x (`AlterColumns`) all resolve to this command through
`BaseResolveHudiAlterTableCommand`, and the schema-on-write ALTER commands
never called it.
- Tests:
- `TestHoodieTableSchemaEvolution` (now on `HoodieCommonTestHarness`):
`testValidateAlterSchemaOnlyChecksSecondaryIndexColumns` builds a table with a
commit and a secondary index on `age`. A rename plus a drop of other columns
fails `validateSchema` with `MissingSchemaFieldException` and passes
`validateAlterSchema`. Retyping `age` with the column drop config on passes
`validateSchema` and fails `validateAlterSchema` with the exact message.
Renaming or dropping `age` fails with the exact message.
`testIndexedColumnMissingFromWriterSchema` pins both values of
`allowMissingColumns`.
- `TestSpark3DDL`: removed `allow.auto.evolution.column.drop=true` from
"Test Chinese table", "Test alter column by add rename and drop", "Test alter
column multiple times" and "Test alter column with complex schema", so they run
under defaults again. New "Test rename and drop column under default write
configs" (COW and MOR): asserts the config is unset, the resulting columns and
rows, one ALTER_SCHEMA commit per DDL, and no pending instant.
- `TestSecondaryIndex` "Column Type Change Should Fail": the same ALTER is
also rejected with the column drop config on; RENAME and DROP of the indexed
column are rejected; no pending instant is left after the rejected ALTERs.
- Removed the same workaround from `TestTimeTravelTable`,
`TestHiveTableSchemaEvolution` (two tests),
`TestNestedSchemaPruningOptimization` and `TestVariantShreddingMixedLayouts`,
with the comments that explained it. `TestVariantShreddingMixedLayouts` is
gated to Spark 4.1, so it compiled but did not run locally (Spark 3.5).
Not changed here: Flink `ALTER TABLE` and the write client's column change
API commit through `BaseHoodieWriteClient#commitTableChange`, which never
called `validateSchema`, so they were not affected by this bug and do not run
the secondary index check either.
### Impact
`ALTER TABLE ... RENAME COLUMN` and `DROP COLUMN` work under default configs
on schema-on-read tables, as before #13595. ALTER TABLE no longer depends on
`hoodie.datasource.write.schema.allow.auto.evolution.column.drop` or
`hoodie.avro.schema.validate`. One visible change for users who set
`hoodie.avro.schema.validate=true`: ALTER TABLE no longer runs the Avro
reader/writer compatibility check, which refused for example `ALTER COLUMN d
TYPE string` on a `date` column, a change schema-on-read supports. Before
#13595 ALTER TABLE did not run it either. `HoodieTable#validateAlterSchema` is
a new public method.
### Risk Level
low. The write path check is unchanged. On the ALTER path the only checks
removed are the missing-field check and the Avro compatibility check described
above; the secondary index check now also runs when the column drop config is
on, and rejects renaming or dropping an indexed column.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]