xiangfu0 opened a new pull request, #19270: URL: https://github.com/apache/pinot/pull/19270
## Summary Adds a **versioned config-migration framework** that transparently upgrades stored `TableConfig`s and `Schema`s to the current version on the controller, so cluster upgrades don't surprise users with configs the new controller can no longer parse or validate. The migration runs as a leader-only controller periodic task. For every table it reads the config *exactly as stored*, runs an ordered migrator chain, and — only if something changed — persists the result through the standard `PinotHelixResourceManager` write path (version-checked write **plus broker/server cache-refresh messages**). It's idempotent: once everything is current, runs are cheap no-ops. ## What's included - **`pinot-spi`** — new `org.apache.pinot.spi.config.migration` package: `ConfigMigrator` / `TableConfigMigrator` / `SchemaMigrator`, `ConfigMigrationRegistry` (dense ordered chain), `MigrationResult`, `ConfigMigrationUtils`. Version markers are **rollback-safe**: `Schema` gains a `configMigrationVersion` field (older readers ignore it via `@JsonIgnoreProperties`), `TableConfig` uses a controller-managed `config.migration.version` custom-config key. - **`pinot-segment-local`** — first concrete migrator (**v0 → v1**) folds deprecated ingestion fields (`tableIndexConfig.streamConfigs`, `segmentsConfig.segmentPushType`/`segmentPushFrequency`) into `ingestionConfig`, finally wiring the previously test-only `TableConfigUtils.convertFromLegacyTableConfig` into a live path. Schema chain ships empty (framework ready; no placeholder transforms). - **`pinot-controller`** — `ConfigMigrationManager` periodic task + wiring in `BaseControllerStarter`; config keys in `ControllerConf`; `CONFIG_MIGRATION_SUCCESS`/`CONFIG_MIGRATION_FAILURE` meters. - **`pinot-common`** — fixes `toTableConfig` to preserve env-var substitution when `applyDecorator=false`; adds a raw-read `getTableConfigWithVersion` overload. ## Configuration | Key | Default | Notes | |---|---|---| | `controller.config.migration.enabled` | `true` | Set `false` to opt out | | `controller.config.migration.frequencyPeriod` | `1h` | | | `controller.config.migration.initialDelaySeconds` | randomized | | | `controller.config.migration.cronExpression` | (none) | | ## Safety / compatibility - **Optimistic concurrency:** version-checked writes never clobber a concurrent operator edit — a lost race is skipped and retried next cycle. - **Validate before persist:** a buggy migrator can never write an invalid config; a not-yet-created schema is treated as transient (skip, no failure metric). - **Rollback-safe:** an older controller ignores the new marker field/key; JSON with the marker + unknown future fields still deserializes. - **Cache convergence:** writes go through the standard RM path so broker/server in-memory caches refresh. ## Tests - **Deprecated → new upgrade** (`pinot-segment-local`): stream-only, batch-only, and combined deprecated configs fold into `ingestionConfig`; deprecated fields cleared; marker stamped; survives a ZK ZNRecord serialization round-trip; already-migrated config is a no-op. - **Registry/markers** (`pinot-spi`): in-order chain, version-compare skip, downgrade left untouched, dense-ordering enforced, TableConfig marker JSON round-trip (preserving user custom configs), Schema marker round-trip + unknown-property tolerance. - **Controller task**: persist-through-RM with version check + success metric, already-current not written, missing-schema skip, persist-failure caught + metered, schema migrated once across hybrid halves. All precommit checks pass (spotless, license, checkstyle: 0 violations) across the four touched modules. ## Notes for reviewers - Enabled **by default** and rewrites stored configs cluster-wide on first upgrade — please confirm this warrants a **release note** and docs for the new `controller.config.migration.*` keys. - Adds a **new public SPI package** and a new `Schema` field — flagging for plugin-maintaining teams (additive, rollback-safe). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
