anshul98ks123 opened a new pull request, #16191: URL: https://github.com/apache/pinot/pull/16191
## Issue [createTableConfigFromOldFormat](https://github.com/startreedata/pinot/blob/5d7fab1cc3177ce21fa2da1e1287f17c461c7df3/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java#L1449) util method fails for a valid tableConfig ## Description In a [recent change](https://github.com/startreedata/pinot/commit/c96c318ef8b9897b04d6f4afe709b1f09fd33cec#diff-ac7ba46a862780f8f703c30869b17418a6d3e6f729e507b18044933a24588438L31-L44), logic for [creating deserializers](https://github.com/startreedata/pinot/blob/5d7fab1cc3177ce21fa2da1e1287f17c461c7df3/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/AbstractIndexType.java#L45) for all IndexTypes was changed to ``` protected ColumnConfigDeserializer<C> createDeserializer() { ColumnConfigDeserializer<C> fromIndexes = IndexConfigDeserializer.fromIndexes(getPrettyName(), getIndexConfigClass()); ColumnConfigDeserializer<C> fromLegacyConfigs = createDeserializerForLegacyConfigs(); return fromLegacyConfigs != null ? fromIndexes.withExclusiveAlternative(fromLegacyConfigs) : fromIndexes; } ``` This creates two deserializers: 1. one for indexes in new format 2. one for legacy format and combined desirealizer has `FAIL` _onConfict_ property, so if both the deserializers detect the same config, it'll result in error like following: ``` Configuration is declared in two different ways for index dictionary on column address ``` example valid table config for which it fails: [example_table_config.json](https://github.com/user-attachments/files/20887301/example_table_config.json) ``` { "tableConfig": { "tableIndexConfig": { ... "varLengthDictionaryColumns": null ... }, ... "fieldConfigList": [ ... { "name": "address", "encodingType": "RAW", "indexTypes": [], "indexes": { "dictionary": { "disabled": false, "onHeap": false, "useVarLengthDictionary": true }, "inverted": { "disabled": false } }, "tierOverwrites": null } ... ] ... }, "schema": { ... } } ``` The logic of `createDeserializerForLegacyConfigs()` for Dictionary index and Json index reads from new format as well, which is incorrect and leads to above mentioned issue Legacy deserializer logic should not infer indexes from new format now This PR fixes that It also adds missing json initializer for VectorIndexConfig -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
