clintropolis opened a new pull request, #20379: URL: https://github.com/apache/druid/pull/20379
### Description As a follow-up to #19830, this PR improves the catalog based compaction to wire catalog stuff up like `baseTable` spec that was missing from the compaction config. `SEALED` catalog tables (with a `baseTable` spec) will enforce the exact schema on any compacted segments, while non-sealed catalog schemas will _always_ analyze the segments to compact to detect any additional columns to extend the catalog schema. `SEALED` will currently only remove columns on tables with a `baseTable` spec defined, but I plan to make it do this universally after #20377 which would allow making an implicit plain table `baseTable` spec from column definitions for `SEALED` tables (meaning a catalog schema for a rollup table or clustered table would always need an explicit base table projection spec defined). A new `CompactionTask` property `sealed` has been added to allow forcing a segment analysis even if a `baseTable` spec is defined, and so the spec for non-sealed tables can be extended with any exist ing columns present in the segment. This PR also fixed up catalog compaction to work properly with MSQ based compaction since the partitioning defined in the catalog wasn't hooked up either. Lastly, documentation has been added for catalog compaction. It will need fixed up after #20377 and its follow-up and the modification to make an implicit non-rollup table when baseTable isn't defined, but that shouldn't be too much work, just need to adjust the section which discussed sealed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
