xiangfu0 opened a new issue, #18808:
URL: https://github.com/apache/pinot/issues/18808
### Feature request
Schema updates currently treat **column deletion as backward-incompatible**.
Removing a column from a table schema is rejected unless the operator passes
`force=true`, which bypasses all validation rather than handling the deletion
safely.
### Current behavior
- `Schema.isBackwardCompatibleWith()`
(`pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java`) returns
`false` if any column present in the old schema is missing from the new one.
- `PinotHelixResourceManager.updateSchema()` throws
`SchemaBackwardIncompatibleException` → HTTP 400 _"Only allow adding new
columns"_, unless `force=true`.
- With `force=true` the check is skipped with only a warning log. The result
is an inconsistent state:
- Existing segments still physically contain the deleted column.
- New segments don't have it.
- Queries referencing the deleted column **silently return null**
(`PinotSegmentColumnReaderFactory`) rather than erroring or being cleaned up.
- There is no `allowDeletedColumn` / managed-deletion config.
### Proposed
Make column deletion a first-class, backward-compatible operation instead of
an all-or-nothing `force` override:
1. Allow column removal through schema update without `force`, behind
explicit intent (e.g. a dedicated flag or validation path that distinguishes
"drop column" from accidental omission).
2. Define + document segment behavior for deleted columns — reclaim the
column on segment reload, and decide query semantics (ignore vs. error on
reference).
3. Preserve the existing guardrails (primary-key changes, type changes) —
only column *removal* is relaxed.
### Notes
- Backward-compatibility / rolling-upgrade sensitive: mixed-version
brokers/servers must agree on how a queried-but-deleted column behaves.
- Related validation: `FieldSpec.isBackwardCompatibleWith()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
