eldenmoon opened a new pull request, #70188:
URL: https://github.com/apache/doris/pull/70188
### What problem does this PR solve?
Issue Number: None
Related PR: #67714 (original; this PR is a reworked, compatibility-first
version of it, and #67714 stays open for comparison)
Problem Summary:
Users cannot add schema template entries or change
`variant_max_subcolumns_count` on an existing VARIANT column without a heavy
schema change. This PR allows both through the light schema change path. Values
written before the change are never cast during compaction.
Commits:
1. [test](regression) Require successful requested compactions: compaction
helpers now fail when compaction is rejected.
2. [fix] Preserve template pattern types in Column identity: MATCH_NAME and
MATCH_NAME_GLOB now take part in equality and hashing.
3. [refactor] Centralize Variant property change validation: moves the
checks into VariantPropertyChangePolicy.
4. [refactor] Share stable template index validation: one helper checks
field_pattern index stability.
5. [refactor] Separate compaction schema pipeline stages: no change to the
output.
6. [feature] Reconcile compaction carriers for added templates: old dynamic
or sparse values under a newly templated path become an untyped carrier and are
not cast.
7. [feature] Keep values when writing added-template carriers: values keep
their JSONB codec, and each path has one destination per segment.
8. [fix] Keep fixed partial update copied VARIANT values off added
templates: values copied from the row store are not re-shredded.
9. [feature] Allow lightweight Variant budget changes: includes an FE replay
test.
10. [test] Cover lightweight Variant budget transitions end to end.
11. [fix] Scan carrier paths that have no physical index.
12. [fix] Do not apply typed predicates to subcolumns stored as another type.
13. [improvement] Keep template type and index when old leaf type matches:
only an exact type match counts.
14. [fix] Keep template carriers out of the dynamic path budget.
15. [improvement] Log the tablet when Variant carrier planning fails.
16. [feature] Allow additive Variant template changes: ADD-only, the old
entries must remain an identical prefix, DOC mode and NestedGroup are rejected,
and DECIMAL256 is rejected.
17. [chore] Remove internal review labels from comments.
Rules:
- Templates are ADD-only. Removal, reorder, rename and type change are
rejected.
- DOC-mode columns reject template and budget changes.
- Field-pattern indexes cannot be added or dropped through this path.
- Both Cloud and row store are supported.
Heavy schema change behavior: after a template ADD, a later heavy schema
change (one that rewrites data) rewrites old values with the template type.
This is intended. Light schema change and compaction never cast old values.
### Release note
Supports adding VARIANT schema template entries and changing
`variant_max_subcolumns_count` with a light schema change.
Compatibility:
- Upgrade all BEs before running a template ALTER.
- After a template ADD, do not roll a BE back to an older version.
### Check List (For Author)
- Test
- [x] Regression test (single-node cluster built from the final head,
fuzzy session variables on): test_variant_modify_max_subcolumns_count,
test_variant_light_alter_matrix, test_variant_template_add_linked_sc,
test_variant_template_fixed_pu_never_altered,
test_variant_template_carrier_budget
- [x] Unit Test: BE filter VariantPathBuilderTest,
VariantWriterHelpersTest, IndexStorage*, SchemaUtilTest, SchemaUtilRowsetTest,
VariantColumnReaderTest, VariantColumnWriterReaderTest,
VariantCompactionPathsTest, VariantDocModeCompactionTest, VariantShredderTest,
VariantRowStoreTest, VariantRowStoreAddedTemplateTest,
FlexibleAddedTemplateVariantTest, FlexiblePartialUpdateTest (372 passed); FE
VariantPropertyChangePolicyTest, ColumnTest, SchemaChangeHandlerTest,
CloudSchemaChangeHandlerTest, VariantColumnNameTest, AlterTableCommandTest,
VariantTemplateIndexBinderTest, InvertedIndexPropertiesTest,
SearchPredicateTest, CreateTableCommandTest, CreateTableInfoTest,
IndexDefinitionTest, OlapTableTest (302 passed). Every commit builds and passes
the BE filter and the FE tests it touches.
- Cloud: covered by UT only, not run on a Cloud cluster.
- [ ] Manual test
- Behavior changed: Yes (new ALTER capability; existing data and behavior
unchanged)
- Does this need documentation: Yes (ALTER for VARIANT templates and budget;
compatibility notes above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
